Skip to content

Bug: circuit breaker can close after fewer than success_threshold probes when an environment wins a new probe election #8645

Closed

Description

@unknownhackerworld

Expected Behaviour

A half-open circuit closes only after success_threshold consecutive probe successes in the current recovery window, as the docs say: "If success_threshold consecutive probes succeed, the circuit closes; a single failing probe reopens it." (docs)

Current Behaviour

The probe success streak is kept in the module-level _LOCAL_SUCCESSES dict (aws_lambda_powertools/utilities/circuit_breaker/base.py). It is only reset when a probe fails in the same environment (_call_probe, except branch) or when this environment closes the circuit. It is never reset when an environment wins a probe election (try_acquire_half_open in handle(), both the OPEN path and the expired-lease takeover path).

So a streak left from an earlier recovery window carries over into the next one. Example with success_threshold=3:

  1. Env A wins the election and gets 2 successful probes (_LOCAL_SUCCESSES["c"] == 2).
  2. Env A goes idle, its probe lease lapses, env B takes over, B's probe fails and the circuit reopens. B runs in another execution environment, so A's counter stays at 2.
  3. After recovery_timeout, env A wins the election again. Its first probe success makes the count 3 and the circuit closes.

The circuit goes back to full traffic after one good probe, while the downstream just failed B's probe. The same thing happens on an expired-lease takeover inside one process: the new owner thread inherits the old owner's streak (noted as pre-existing in #8323).

Code snippet

Runs from the repo root on develop (uses the in-memory FakePersistence from the test suite, no AWS access needed):

import datetime as dt

from aws_lambda_powertools.utilities.circuit_breaker import CircuitBreakerConfig, CircuitState, circuit_breaker
from aws_lambda_powertools.utilities.circuit_breaker.persistence.record import CircuitStateRecord
from tests.functional.circuit_breaker.conftest import FakePersistence

now = int(dt.datetime.now().timestamp())
store = FakePersistence()
config = CircuitBreakerConfig(recovery_timeout=30, success_threshold=3, local_cache_max_age=0)
store.db["c"] = CircuitStateRecord(name="c", state=CircuitState.OPEN, failure_count=5, opened_at=now - 60)


@circuit_breaker(name="c", persistence_store=store, config=config)
def call():
    return "ok"


# This environment wins the probe election and gets 2 of the 3 successes it needs.
call()
call()
print(store.db["c"].state)  # HALF_OPEN

# Its lease lapses, another environment takes over, that probe fails and the circuit reopens.
store.db["c"] = CircuitStateRecord(name="c", state=CircuitState.OPEN, failure_count=0, opened_at=now - 60)

# Next recovery window: this environment wins again and its first probe succeeds.
call()
print(store.db["c"].state)  # CLOSED after a single probe, expected HALF_OPEN (1 of 3)

Output:

HALF_OPEN
CLOSED

Possible Solution

Reset _LOCAL_SUCCESSES[name] to 0 (under _COUNTERS_LOCK) right after this worker wins try_acquire_half_open, on both the OPEN path and the expired-lease takeover path. Winning an election starts a new probe streak, so nothing from an earlier window or an earlier owner should count. The owner continuing its own lease does not go through the election, so its streak is kept. This mirrors how _LAST_OBSERVED_STATE already invalidates a stale failure streak after an external close.

I'd like to work on this. I have the fix and regression tests ready and will open a PR linked to this issue.

Steps to Reproduce

Run the snippet above from the repository root.

Powertools for AWS Lambda (Python) version

latest (3.36.0, also on develop at 31180f0)

AWS Lambda function runtime

3.13

Packaging format used

PyPi

Debugging logs

N/A

Activity

  1. boring-cyborg commented on Oct 20, 2026

    @boring-cyborg

    Thanks for opening your first issue here! We'll come back to you as soon as we can.
    In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

  2. powertools-for-aws-oss-automation commented on Oct 20, 2026

    @powertools-for-aws-oss-automation

    Thanks for opening your first issue here! We'll come back to you as soon as we can.
    In the meantime, check out our GitHub Discussions: Join the conversation

  3. added a commit that references this issue on Oct 20, 2026
    2764509
  4. leandrodamascena commented on Oct 20, 2026

    @leandrodamascena
    Contributor

    Hey @unknownhackerworld, thanks a lot for this one, great catch and a really clear write-up! I confirmed it running multiple environments against a real DynamoDB table, the circuit closed early a few times exactly like you described, and with your reset it didn't happen anymore. Pls go ahead with the PR, happy to review it.

  5. powertools-for-aws-oss-automation commented on Oct 20, 2026

    @powertools-for-aws-oss-automation

    Warning

    This issue is now closed. Please be mindful that future comments are hard for our team to see.
    If you need more assistance, please either reopen the issue, or open a new issue referencing this one.
    If you wish to keep having a conversation with other community members under this issue feel free to do so.

  6. added a commit that references this issue on Oct 20, 2026
    4fe0d6d
  7. github-actions commented on Oct 22, 2026

    @github-actions
    Contributor

    This is now released under 3.37.0 version!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions