Repository navigation
test(sysdialog): the named-secret test gets an empty secret list, so it passes when run again - #758
Merged
Merged
Conversation
…it passes when run again
jay23606
added a commit
to jay23606/flutter-probe
that referenced
this pull request
Nov 17, 2026
…it passes when run again (AlphaWaveSystems#758) ## Summary I looked for timing-fragile tests in `internal/` by running the whole tree under load: `go test -race -count=3 ./internal/...`, with one `yes > /dev/null` per CPU (18 processes) for the whole run, on macOS (darwin/arm64). Only one test failed, `TestNoteNamedSecretMasksByNameOnly` in `internal/sysdialog`, in 2 of its 3 runs. Its cause is shared state, not the clock. It fails the same way under plain `go test -count=2`, with no load. **Cause.** `NoteNamedSecret` adds a value to the secret list, which lives for the whole process. The test checks that the text changes from before to after the value is noted. On the second run of the test, the value is already in the list before it is noted, so nothing changes and every case that should be masked reports a failure: ``` password="hunter2-page-Zq": masked = true, want true ("****") ``` **Fix.** A test helper, `emptyKnownSecrets(t)`, gives the test an empty secret list and restores the old list when the test ends. The test now checks the same thing on every run. No product code changes, so there is no CHANGELOG entry. ## Results | Run | Result | |---|---| | Before: `go test -race -count=3 ./internal/...` under load | 1 failing test: `TestNoteNamedSecretMasksByNameOnly`, 2 of 3 runs. Every other package `ok`. | | Before: `go test -count=2 -run TestNoteNamedSecretMasksByNameOnly$ ./internal/sysdialog/` without load | FAIL | | After: `go test -race -count=20` of the test | ok | | After: `go test -race -count=3 ./internal/sysdialog/` | ok | ## Mutation check Each mutant was applied, the test was run with `-count=2`, and the file was restored byte for byte. | Mutant | Result | |---|---| | `NoteNamedSecret` notes nothing | killed | | the test does not call the helper (the original bug) | killed | | the helper keeps the old list instead of emptying it | survives: equivalent for this bug | The third mutant survives because the helper does two things, and either one alone fixes this bug. Restoring the old list at the end drops what the run added, and emptying the list at the start makes the next run clean. Both stay. Emptying keeps the test independent of values that other tests note, and restoring keeps other tests independent of this one. ## Timing-sensitive tests that did not fail (listed, not changed) None of these failed in the 3 runs under load above. - About 30 real-clock upper bounds on how fast something must fail or finish. Most are 2 to 10 s, for example in `internal/probelink`, `internal/httpcall`, `internal/device` (webview session, cdp, crash collect) and `internal/runner` (timeouts, reconnect). - The tightest bounds: - `internal/runner` `TestWaitAtOrAboveTheLimitFailsAtOnce`, under 500 ms. - `internal/runner/test_timeout_pins_test.go:189`, a window of 900 ms to 2.5 s. - 52 `time.Sleep` calls in tests across `internal/`. The wait-rule race in `internal/runner` was fixed in AlphaWaveSystems#750. Also noticed: `gofmt -l` lists `internal/sysdialog/roles.go`. That file is the same as on main, so it is not part of this change. ## Test plan - [x] Load run before the fix (table above). - [x] The test and its package after the fix (table above). - [x] Mutation check (table above). - [x] `gofmt` clean on the changed file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I looked for timing-fragile tests in
internal/by running the whole tree under load:go test -race -count=3 ./internal/..., with oneyes > /dev/nullper CPU (18 processes) for the whole run, on macOS (darwin/arm64).Only one test failed,
TestNoteNamedSecretMasksByNameOnlyininternal/sysdialog, in 2 of its 3 runs. Its cause is shared state, not the clock. It fails the same way under plaingo test -count=2, with no load.Cause.
NoteNamedSecretadds a value to the secret list, which lives for the whole process. The test checks that the text changes from before to after the value is noted. On the second run of the test, the value is already in the list before it is noted, so nothing changes and every case that should be masked reports a failure:Fix. A test helper,
emptyKnownSecrets(t), gives the test an empty secret list and restores the old list when the test ends. The test now checks the same thing on every run. No product code changes, so there is no CHANGELOG entry.Results
go test -race -count=3 ./internal/...under loadTestNoteNamedSecretMasksByNameOnly, 2 of 3 runs. Every other packageok.go test -count=2 -run TestNoteNamedSecretMasksByNameOnly$ ./internal/sysdialog/without loadgo test -race -count=20of the testgo test -race -count=3 ./internal/sysdialog/Mutation check
Each mutant was applied, the test was run with
-count=2, and the file was restored byte for byte.NoteNamedSecretnotes nothingThe third mutant survives because the helper does two things, and either one alone fixes this bug. Restoring the old list at the end drops what the run added, and emptying the list at the start makes the next run clean. Both stay. Emptying keeps the test independent of values that other tests note, and restoring keeps other tests independent of this one.
Timing-sensitive tests that did not fail (listed, not changed)
None of these failed in the 3 runs under load above.
internal/probelink,internal/httpcall,internal/device(webview session, cdp, crash collect) andinternal/runner(timeouts, reconnect).internal/runnerTestWaitAtOrAboveTheLimitFailsAtOnce, under 500 ms.internal/runner/test_timeout_pins_test.go:189, a window of 900 ms to 2.5 s.time.Sleepcalls in tests acrossinternal/.The wait-rule race in
internal/runnerwas fixed in #750.Also noticed:
gofmt -llistsinternal/sysdialog/roles.go. That file is the same as on main, so it is not part of this change.Test plan
gofmtclean on the changed file.