Skip to content

test(sysdialog): the named-secret test gets an empty secret list, so it passes when run again - #758

Merged
pbertsch merged 1 commit into
mainfrom
test/timing-fragile
Nov 17, 2026
Merged

pbertsch merged 1 commit into
mainfrom
test/timing-fragile

Conversation

@pbertsch

Copy link
Copy Markdown
Member

Summary

I looked for timing-fragile tests in internal/ by running the whole tree under load: go test -race -count=3 ./internal/..., with one yes > /dev/null per CPU (18 processes) for the whole run, on macOS (darwin/arm64).

Only one test failed, TestNoteNamedSecretMasksByNameOnly in internal/sysdialog, in 2 of its 3 runs. Its cause is shared state, not the clock. It fails the same way under plain go test -count=2, with no load.

Cause. NoteNamedSecret adds a value to the secret list, which lives for the whole process. The test checks that the text changes from before to after the value is noted. On the second run of the test, the value is already in the list before it is noted, so nothing changes and every case that should be masked reports a failure:

password="hunter2-page-Zq": masked = true, want true ("****")

Fix. A test helper, emptyKnownSecrets(t), gives the test an empty secret list and restores the old list when the test ends. The test now checks the same thing on every run. No product code changes, so there is no CHANGELOG entry.

Results

Run Result
Before: go test -race -count=3 ./internal/... under load 1 failing test: TestNoteNamedSecretMasksByNameOnly, 2 of 3 runs. Every other package ok.
Before: go test -count=2 -run TestNoteNamedSecretMasksByNameOnly$ ./internal/sysdialog/ without load FAIL
After: go test -race -count=20 of the test ok
After: go test -race -count=3 ./internal/sysdialog/ ok

Mutation check

Each mutant was applied, the test was run with -count=2, and the file was restored byte for byte.

Mutant Result
NoteNamedSecret notes nothing killed
the test does not call the helper (the original bug) killed
the helper keeps the old list instead of emptying it survives: equivalent for this bug

The third mutant survives because the helper does two things, and either one alone fixes this bug. Restoring the old list at the end drops what the run added, and emptying the list at the start makes the next run clean. Both stay. Emptying keeps the test independent of values that other tests note, and restoring keeps other tests independent of this one.

Timing-sensitive tests that did not fail (listed, not changed)

None of these failed in the 3 runs under load above.

  • About 30 real-clock upper bounds on how fast something must fail or finish. Most are 2 to 10 s, for example in internal/probelink, internal/httpcall, internal/device (webview session, cdp, crash collect) and internal/runner (timeouts, reconnect).
  • The tightest bounds:
    • internal/runner TestWaitAtOrAboveTheLimitFailsAtOnce, under 500 ms.
    • internal/runner/test_timeout_pins_test.go:189, a window of 900 ms to 2.5 s.
  • 52 time.Sleep calls in tests across internal/.

The wait-rule race in internal/runner was fixed in #750.

Also noticed: gofmt -l lists internal/sysdialog/roles.go. That file is the same as on main, so it is not part of this change.

Test plan

  • Load run before the fix (table above).
  • The test and its package after the fix (table above).
  • Mutation check (table above).
  • gofmt clean on the changed file.

@pbertsch
pbertsch requested a review from a team as a code owner November 17, 2026 04:46
@pbertsch
pbertsch merged commit 30cf7bf into main Nov 17, 2026
10 of 11 checks passed
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…it passes when run again (AlphaWaveSystems#758)

## Summary

I looked for timing-fragile tests in `internal/` by running the whole tree under load: `go test -race -count=3 ./internal/...`, with one `yes > /dev/null` per CPU (18 processes) for the whole run, on macOS (darwin/arm64).

Only one test failed, `TestNoteNamedSecretMasksByNameOnly` in `internal/sysdialog`, in 2 of its 3 runs. Its cause is shared state, not the clock. It fails the same way under plain `go test -count=2`, with no load.

**Cause.** `NoteNamedSecret` adds a value to the secret list, which lives for the whole process. The test checks that the text changes from before to after the value is noted. On the second run of the test, the value is already in the list before it is noted, so nothing changes and every case that should be masked reports a failure:

```
password="hunter2-page-Zq": masked = true, want true ("****")
```

**Fix.** A test helper, `emptyKnownSecrets(t)`, gives the test an empty secret list and restores the old list when the test ends. The test now checks the same thing on every run. No product code changes, so there is no CHANGELOG entry.

## Results

| Run | Result |
|---|---|
| Before: `go test -race -count=3 ./internal/...` under load | 1 failing test: `TestNoteNamedSecretMasksByNameOnly`, 2 of 3 runs. Every other package `ok`. |
| Before: `go test -count=2 -run TestNoteNamedSecretMasksByNameOnly$ ./internal/sysdialog/` without load | FAIL |
| After: `go test -race -count=20` of the test | ok |
| After: `go test -race -count=3 ./internal/sysdialog/` | ok |

## Mutation check

Each mutant was applied, the test was run with `-count=2`, and the file was restored byte for byte.

| Mutant | Result |
|---|---|
| `NoteNamedSecret` notes nothing | killed |
| the test does not call the helper (the original bug) | killed |
| the helper keeps the old list instead of emptying it | survives: equivalent for this bug |

The third mutant survives because the helper does two things, and either one alone fixes this bug. Restoring the old list at the end drops what the run added, and emptying the list at the start makes the next run clean. Both stay. Emptying keeps the test independent of values that other tests note, and restoring keeps other tests independent of this one.

## Timing-sensitive tests that did not fail (listed, not changed)

None of these failed in the 3 runs under load above.

- About 30 real-clock upper bounds on how fast something must fail or finish. Most are 2 to 10 s, for example in `internal/probelink`, `internal/httpcall`, `internal/device` (webview session, cdp, crash collect) and `internal/runner` (timeouts, reconnect).
- The tightest bounds:
  - `internal/runner` `TestWaitAtOrAboveTheLimitFailsAtOnce`, under 500 ms.
  - `internal/runner/test_timeout_pins_test.go:189`, a window of 900 ms to 2.5 s.
- 52 `time.Sleep` calls in tests across `internal/`.

The wait-rule race in `internal/runner` was fixed in AlphaWaveSystems#750.

Also noticed: `gofmt -l` lists `internal/sysdialog/roles.go`. That file is the same as on main, so it is not part of this change.

## Test plan

- [x] Load run before the fix (table above).
- [x] The test and its package after the fix (table above).
- [x] Mutation check (table above).
- [x] `gofmt` clean on the changed file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant