Repository navigation
Description
Expected Behaviour
A half-open circuit closes only after success_threshold consecutive probe successes in the current recovery window, as the docs say: "If success_threshold consecutive probes succeed, the circuit closes; a single failing probe reopens it." (docs)
Current Behaviour
The probe success streak is kept in the module-level _LOCAL_SUCCESSES dict (aws_lambda_powertools/utilities/circuit_breaker/base.py). It is only reset when a probe fails in the same environment (_call_probe, except branch) or when this environment closes the circuit. It is never reset when an environment wins a probe election (try_acquire_half_open in handle(), both the OPEN path and the expired-lease takeover path).
So a streak left from an earlier recovery window carries over into the next one. Example with success_threshold=3:
- Env A wins the election and gets 2 successful probes (
_LOCAL_SUCCESSES["c"] == 2). - Env A goes idle, its probe lease lapses, env B takes over, B's probe fails and the circuit reopens. B runs in another execution environment, so A's counter stays at 2.
- After
recovery_timeout, env A wins the election again. Its first probe success makes the count 3 and the circuit closes.
The circuit goes back to full traffic after one good probe, while the downstream just failed B's probe. The same thing happens on an expired-lease takeover inside one process: the new owner thread inherits the old owner's streak (noted as pre-existing in #8323).
Code snippet
Runs from the repo root on develop (uses the in-memory FakePersistence from the test suite, no AWS access needed):
import datetime as dt
from aws_lambda_powertools.utilities.circuit_breaker import CircuitBreakerConfig, CircuitState, circuit_breaker
from aws_lambda_powertools.utilities.circuit_breaker.persistence.record import CircuitStateRecord
from tests.functional.circuit_breaker.conftest import FakePersistence
now = int(dt.datetime.now().timestamp())
store = FakePersistence()
config = CircuitBreakerConfig(recovery_timeout=30, success_threshold=3, local_cache_max_age=0)
store.db["c"] = CircuitStateRecord(name="c", state=CircuitState.OPEN, failure_count=5, opened_at=now - 60)
@circuit_breaker(name="c", persistence_store=store, config=config)
def call():
return "ok"
# This environment wins the probe election and gets 2 of the 3 successes it needs.
call()
call()
print(store.db["c"].state) # HALF_OPEN
# Its lease lapses, another environment takes over, that probe fails and the circuit reopens.
store.db["c"] = CircuitStateRecord(name="c", state=CircuitState.OPEN, failure_count=0, opened_at=now - 60)
# Next recovery window: this environment wins again and its first probe succeeds.
call()
print(store.db["c"].state) # CLOSED after a single probe, expected HALF_OPEN (1 of 3)Output:
HALF_OPEN
CLOSED
Possible Solution
Reset _LOCAL_SUCCESSES[name] to 0 (under _COUNTERS_LOCK) right after this worker wins try_acquire_half_open, on both the OPEN path and the expired-lease takeover path. Winning an election starts a new probe streak, so nothing from an earlier window or an earlier owner should count. The owner continuing its own lease does not go through the election, so its streak is kept. This mirrors how _LAST_OBSERVED_STATE already invalidates a stale failure streak after an external close.
I'd like to work on this. I have the fix and regression tests ready and will open a PR linked to this issue.
Steps to Reproduce
Run the snippet above from the repository root.
Powertools for AWS Lambda (Python) version
latest (3.36.0, also on develop at 31180f0)
AWS Lambda function runtime
3.13
Packaging format used
PyPi
Debugging logs
N/AMetadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsComing soon
Activity
boring-cyborg commentedon Oct 20, 2026
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link
powertools-for-aws-oss-automation commentedon Oct 20, 2026
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out our GitHub Discussions: Join the conversation
fix(circuit_breaker): start a fresh probe success streak on each prob…
leandrodamascena commentedon Oct 20, 2026
Hey @unknownhackerworld, thanks a lot for this one, great catch and a really clear write-up! I confirmed it running multiple environments against a real DynamoDB table, the circuit closed early a few times exactly like you described, and with your reset it didn't happen anymore. Pls go ahead with the PR, happy to review it.
powertools-for-aws-oss-automation commentedon Oct 20, 2026
Warning
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either reopen the issue, or open a new issue referencing this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
fix(circuit_breaker): start a fresh probe success streak on each prob…
github-actions commentedon Oct 22, 2026
This is now released under 3.37.0 version!