Skip to content

fix(agent): one visible text is one match: see exactly no longer counts a Text twice (issue #800) - #806

Merged
pbertsch merged 2 commits into
mainfrom
fix/agent-count-text-once
Nov 17, 2026
Merged

pbertsch merged 2 commits into
mainfrom
fix/agent-count-text-once

Conversation

@pbertsch

@pbertsch pbertsch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #800: see exactly N "<text>" counted an ordinary Text twice.

Cause. In probe_agent/lib/src/finder.dart, _matchesText matches a Text by its data, and also the RichText that Text builds internally, by toPlainText(). Both elements were returned for one visible text, so:

  • see exactly 1 found 2;
  • an ordinal (2nd "Row") could pick the first Text's own inner RichText;
  • a text inside a container (see "x" in "y") was doubled.

Fix. _oncePerText runs on the matches of the text search and of the container search. It leaves out a RichText whose nearest Text above it, the one that built it, is also in the matches. The Text stays; its render object is the same paragraph. These are kept:

  • a RichText of its own;
  • a Text.rich, which matches only through its RichText;
  • a Text and a separate Text.rich with the same words (two texts, two matches).

Breaking for a suite that asserted the doubled count: see exactly 2 for one Text now fails.

Tests

probe_agent/test/finder_count_test.dart (9 tests):

  • a plain Text is one match, and so is a part of it;
  • a Text.rich, a RichText and a button's label are each one match;
  • two equal Texts are 2;
  • 2nd "Row" is the second Text, not the first one's RichText;
  • in a container, the Text and its RichText are one match;
  • a Text and a Text.rich with the same words are 2.

Five of these failed on main's code: a Text 2, its part 2, a button 2, two Texts 4, and the ordinal picking the inner one.

probe_agent/test/executor_see_exactly_test.dart (3 tests): see exactly 1 on one Text passes and exactly 2 fails with "found 1"; two equal Texts pass exactly 2; a button's label counts once.

All 261 probe_agent tests pass.

Mutation check

Each mutant was applied, the tests were run, and the file was restored byte for byte. All 4 were killed. Two survived at first, because the container test found nothing and a lone Text.rich never reached the check; the two tests above were added for them.

Mutant Result
the text search is not deduped killed
the container search is not deduped killed
every RichText is dropped killed
the Text is dropped instead of its RichText killed

Device check

Compat-Pixel-API35 (Android 15, API 35, emulator-5590) and Compat-iPhone (iPhone 17 Pro, iOS 26.5, UDID A57AEE67-3473-4C20-BE25-8847C2A9D6BE); Flutter 3.44.8, the buttons screen. Three tests: see exactly 1 "Press elevated", see exactly 1 "Elevated: 0", and see exactly 2 "Press elevated" (which must fail):

Agent exactly 1 (label) exactly 1 (state text) exactly 2
without this fix (Android) fails: "found 2" fails: "found 2" passes
this PR (Android) passes passes fails: "found 1"
this PR (iOS) passes passes fails: "found 1"

Docs

  • probescript/dictionary.md: what see exactly counts.
  • tools/agent.md: "one visible text is one match".
  • docs/compat/MATRIX.md: the Lists finding is fixed in the next probe_agent release.
  • CHANGELOG.md and probe_agent/CHANGELOG.md, Fixed, marked breaking.

The Lists negative variant N9 (probe-tests/negative/lists.probe.expect-fail) pins the 0.27.0 behaviour. It will pass once an agent with this fix runs, so the Lists screen's owner updates it when this lands; that's coordinated.

No version bump: it happens at release time, per the checklist.

Test plan

  • flutter test in probe_agent: 261 passed.
  • go test ./scripts/: the negative-suite guards pass.
  • Device check above, on both named devices.

Review follow-up

  • Breaking, ordinals too. Both CHANGELOGs and tools/agent.md now also say that an ordinal which landed on the inner paragraph of a single Text (tap 2nd "8" where only one "8" is shown) now finds nothing; write tap "8".

  • The example suites tests/login.probe, tests/cart.probe and tests/recipes/login.probe use tap 2nd "<digit>" on a keypad. They belong to an old example app (an activation-code keypad) that isn't in this repository. No workflow, gate or docs page runs them; the READMEs' tests/login.probe example doesn't use 2nd. Whether they relied on the quirk depends on that app's screen:

    • if a digit is shown once, the old 2nd found that key's own inner paragraph, and it now fails;
    • if a display also shows the digit after a tap, the old 2nd found the display's inner paragraph (a wrong target), and it's now the key.

    I couldn't run them, so I left them unchanged rather than guess, and flag them here.

  • The mixed-screen test (a Text and a Text.rich with the same words, 2 matches) is marked in a comment as the key guard: a rule that dropped every RichText, or none, fails it.

  • N9 of lists.probe.expect-fail: its owner flips it once this merges.

@pbertsch
pbertsch requested a review from a team as a code owner November 17, 2026 14:33
@pbertsch
pbertsch merged commit 2faa4b5 into main Nov 17, 2026
10 of 12 checks passed
pbertsch added a commit that referenced this pull request Nov 17, 2026
…tension tests for ProbeScript doc changes (#815)

## Summary

Main is red on two VS Code extension checks: "the generated keyword files are up to date with the docs" and "the generated snippet files are up to date with the docs". #806 changed the `see exactly` entry in `website/src/content/docs/probescript/dictionary.md` and didn't regenerate the extension's files from it.

- **Regenerated** with `npm run gen:keywords` and `npm run gen:snippets` in `vscode/`. The diff is one line in each file: the `see exactly` snippet description, and the docs hash in `probescriptKeywords.generated.ts`.
- **Why it wasn't caught:** `scripts/ci-local.sh` adds the Studio frontend check to a default run when the change touches the ProbeScript docs. It didn't add the VS Code extension tests, which are made from the same docs, and the hosted workflow only runs for changes under `studio/` and `vscode/`. Both checks are now added for those paths, and the run's note says so. `--all` and `--job` are unchanged.

No CHANGELOG entry: generated files and local CI tooling only, with nothing user-facing.

## Test plan

- [x] `npm test` in `vscode/`: 84 of 84 pass (the two checks fail on main).
- [x] `go test ./scripts/` passes. `ci_local_plan_test.go` now expects both checks, and the note, for each docs input. It expects neither check for other paths, and each check once under `--all` with no note.
- [x] Mutant: adding only the frontend, as before, fails the plan test.
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…ts a Text twice (issue AlphaWaveSystems#800) (AlphaWaveSystems#806)

## Summary

Fixes AlphaWaveSystems#800: `see exactly N "<text>"` counted an ordinary `Text` twice.

**Cause.** In `probe_agent/lib/src/finder.dart`, `_matchesText` matches a `Text` by its `data`, and also the `RichText` that `Text` builds internally, by `toPlainText()`. Both elements were returned for one visible text, so:
- `see exactly 1` found 2;
- an ordinal (`2nd "Row"`) could pick the first `Text`'s own inner `RichText`;
- a text inside a container (`see "x" in "y"`) was doubled.

**Fix.** `_oncePerText` runs on the matches of the text search and of the container search. It leaves out a `RichText` whose nearest `Text` above it, the one that built it, is also in the matches. The `Text` stays; its render object is the same paragraph. These are kept:
- a `RichText` of its own;
- a `Text.rich`, which matches only through its `RichText`;
- a `Text` and a separate `Text.rich` with the same words (two texts, two matches).

**Breaking** for a suite that asserted the doubled count: `see exactly 2` for one `Text` now fails.

## Tests

`probe_agent/test/finder_count_test.dart` (9 tests):
- a plain `Text` is one match, and so is a part of it;
- a `Text.rich`, a `RichText` and a button's label are each one match;
- two equal `Text`s are 2;
- `2nd "Row"` is the second `Text`, not the first one's `RichText`;
- in a container, the `Text` and its `RichText` are one match;
- a `Text` and a `Text.rich` with the same words are 2.

Five of these failed on `main`'s code: a `Text` 2, its part 2, a button 2, two `Text`s 4, and the ordinal picking the inner one.

`probe_agent/test/executor_see_exactly_test.dart` (3 tests): `see exactly 1` on one `Text` passes and `exactly 2` fails with "found 1"; two equal `Text`s pass `exactly 2`; a button's label counts once.

All 261 `probe_agent` tests pass.

## Mutation check

Each mutant was applied, the tests were run, and the file was restored byte for byte. All 4 were killed. Two survived at first, because the container test found nothing and a lone `Text.rich` never reached the check; the two tests above were added for them.

| Mutant | Result |
|---|---|
| the text search is not deduped | killed |
| the container search is not deduped | killed |
| every `RichText` is dropped | killed |
| the `Text` is dropped instead of its `RichText` | killed |

## Device check

`Compat-Pixel-API35` (Android 15, API 35, `emulator-5590`) and `Compat-iPhone` (iPhone 17 Pro, iOS 26.5, UDID `A57AEE67-3473-4C20-BE25-8847C2A9D6BE`); Flutter 3.44.8, the `buttons` screen. Three tests: `see exactly 1 "Press elevated"`, `see exactly 1 "Elevated: 0"`, and `see exactly 2 "Press elevated"` (which must fail):

| Agent | `exactly 1` (label) | `exactly 1` (state text) | `exactly 2` |
|---|---|---|---|
| without this fix (Android) | **fails: "found 2"** | **fails: "found 2"** | passes |
| this PR (Android) | passes | passes | fails: "found 1" |
| this PR (iOS) | passes | passes | fails: "found 1" |

## Docs

- `probescript/dictionary.md`: what `see exactly` counts.
- `tools/agent.md`: "one visible text is one match".
- `docs/compat/MATRIX.md`: the Lists finding is fixed in the next `probe_agent` release.
- `CHANGELOG.md` and `probe_agent/CHANGELOG.md`, Fixed, marked breaking.

**The Lists negative variant N9** (`probe-tests/negative/lists.probe.expect-fail`) pins the 0.27.0 behaviour. It will pass once an agent with this fix runs, so the Lists screen's owner updates it when this lands; that's coordinated.

No version bump: it happens at release time, per the checklist.

## Test plan

- [x] `flutter test` in `probe_agent`: 261 passed.
- [x] `go test ./scripts/`: the negative-suite guards pass.
- [x] Device check above, on both named devices.

## Review follow-up

- **Breaking, ordinals too.** Both CHANGELOGs and `tools/agent.md` now also say that an ordinal which landed on the inner paragraph of a single `Text` (`tap 2nd "8"` where only one "8" is shown) now finds nothing; write `tap "8"`.
- **The example suites** `tests/login.probe`, `tests/cart.probe` and `tests/recipes/login.probe` use `tap 2nd "<digit>"` on a keypad. They belong to an old example app (an activation-code keypad) that isn't in this repository. No workflow, gate or docs page runs them; the READMEs' `tests/login.probe` example doesn't use `2nd`. Whether they relied on the quirk depends on that app's screen:
  - if a digit is shown once, the old `2nd` found that key's own inner paragraph, and it now fails;
  - if a display also shows the digit after a tap, the old `2nd` found the *display's* inner paragraph (a wrong target), and it's now the key.

  I couldn't run them, so I left them unchanged rather than guess, and flag them here.
- **The mixed-screen test** (a `Text` and a `Text.rich` with the same words, 2 matches) is marked in a comment as the key guard: a rule that dropped every `RichText`, or none, fails it.
- **N9 of `lists.probe.expect-fail`:** its owner flips it once this merges.
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…tension tests for ProbeScript doc changes (AlphaWaveSystems#815)

## Summary

Main is red on two VS Code extension checks: "the generated keyword files are up to date with the docs" and "the generated snippet files are up to date with the docs". AlphaWaveSystems#806 changed the `see exactly` entry in `website/src/content/docs/probescript/dictionary.md` and didn't regenerate the extension's files from it.

- **Regenerated** with `npm run gen:keywords` and `npm run gen:snippets` in `vscode/`. The diff is one line in each file: the `see exactly` snippet description, and the docs hash in `probescriptKeywords.generated.ts`.
- **Why it wasn't caught:** `scripts/ci-local.sh` adds the Studio frontend check to a default run when the change touches the ProbeScript docs. It didn't add the VS Code extension tests, which are made from the same docs, and the hosted workflow only runs for changes under `studio/` and `vscode/`. Both checks are now added for those paths, and the run's note says so. `--all` and `--job` are unchanged.

No CHANGELOG entry: generated files and local CI tooling only, with nothing user-facing.

## Test plan

- [x] `npm test` in `vscode/`: 84 of 84 pass (the two checks fail on main).
- [x] `go test ./scripts/` passes. `ci_local_plan_test.go` now expects both checks, and the note, for each docs input. It expects neither check for other paths, and each check once under `--all` with no note.
- [x] Mutant: adding only the frontend, as before, fails the plan test.
pbertsch added a commit that referenced this pull request Nov 17, 2026
…erts the wrong count (#821)

## Summary

Follow-up of #798 and #806. The agent of main now counts one visible `Text` once (issue #800, #806), so negative variant N9 of the Lists suite, which pinned the old double count, would pass. This flips it as agreed:

- `lists.probe` has a new test, `see exactly counts a visible Text once`: `see exactly 1` of a row title, a state text and a button label (12 tests).
- N9 of `lists.probe.expect-fail` now asserts the wrong count: `see exactly 2 "Row 0"` must fail with "found 1".
- `docs/compat/MATRIX.md` and CHANGELOG say that the finding is fixed and what the suite does now.

## Checked

- Device: emulator-5600, AVD `ProbeWebViewSpike` (name read back before each run); the fixture is built with the agent of this branch (a path dependency), so it has the fix.
  - **Flutter 3.50.0**: `lists.probe` **12 of 12**; negative file **0 passed, 9 failed**, N9 at its own step with `Expected exactly 2 "text("Row 0")" but found 1`.
  - **Flutter 3.44.8**: `lists.probe` in 3 runs, 11 of 12 in each, the twelfth test not running because `before each test` failed to restart the app ("reconnect with pre-shared token: agent not reachable within 20s"), a different test each time; every test passed in at least one run, but not all 12 in one run. Negative file in 3 runs: 0 passed, 9 failed in each; in 2 of the 3 runs one test failed in `before each test` (the same restart failure) instead of at its own step, and in the third run every test failed at its own step with N9 ending in "found 1".
- `make ci-local`: 5 of 5; disclosure scan clean.

## Not checked / limits

- The restart failure is tracked in issue #823 (measurements per Flutter build, the `probe.yaml` of the runs; the machine load was not measured): in the runs above it hit every full run on 3.44.8, and 12 of 12 was reached on 3.50.0 in the runs made at the time; later runs on 3.50.0 (made for another pull request) had the same failure, so it is not specific to one Flutter version.
- One device, one Android version; no iOS.
pbertsch added a commit that referenced this pull request Nov 17, 2026
…xts, rich_texts, max, counts, truncated) (#837)

## Summary

The agent change from `docs/proposals/studio-replace-line-step.md`, item 3 of its plan. Studio's "Replace the line" offer may be shown only when the visible snapshot is provably complete, and the `probe.visible_summary` answer couldn't say that. It now carries additive keys. An older CLI ignores them: `probelink`'s `VisibleSummaryResult` decodes only `texts` and `keys`, and the Go tests still pass.

| Key | What it holds |
|---|---|
| `editable_texts` | the value of every visible field that isn't obscured, recorded **independently of the de-duplication** of `texts`. Before, a `Text` "Sign In" followed by a field holding "Sign In" left one entry, and the field, which a text selector also matches, couldn't be seen. |
| `rich_texts` | the texts of visible `RichText`s that no `Text` built. A text selector matches them, but they weren't in the summary. A `Text`'s own paragraph isn't listed twice; the check is the nearest `Text` above, with no other `RichText` between, as in #806. |
| `max` | the cap of each list (25) |
| `counts` | distinct items each list saw, before the cut |
| `truncated` | `true` when any list saw more than `max` |

- **Unchanged:** `texts` and `keys` keep their content and order. `visibleSummary`, used by the failure hints and the screen fingerprint, now projects the two lists from the same walk.
- **One walk:** a single walk collects every distinct item in tree order, then each list is cut at `max`.
- **Obscured fields:** an obscured field adds nothing to any list.
- **No length cut in the agent:** the agent never leaves an item out for its length. The 40-character rule in the spec is Go's (`wholeItems` counts such an item in `Cut`), so the agent test checks that a 43-character text arrives whole.
- **Not in this PR:** reading the keys in Go (`probelink`, `runner.VisibleItems`) and Studio's offer. That's the Studio side of the spec, and per the spec it needs a published agent release.

## Tests (`probe_agent/test/visible_summary_full_test.dart`, 9)

- **De-duplication:** a `Text` "Sign In" and a field holding "Sign In" give `texts` `["Sign In"]` and `editable_texts` `["Sign In"]`, with counts and `truncated: false`.
- **Obscured fields:** the obscured value appears nowhere in the JSON answer; the plain field is listed.
- **Raw `RichText`:** a raw `RichText` is in `rich_texts`. A plain `Text` and a `Text.rich` stay only in `texts`.
- **24, 25 and 26 distinct texts:** the list holds 24, 25 and 25; `counts.texts` is 24, 25 and 26; `truncated` is false, false, true.
- **Long text:** a 43-character text is listed whole.
- **Fields over the cap:** 26 fields with `max: 26` aren't truncated; with `max: 25` the list holds 25 and `truncated` is true.
- **Failure hints:** `visibleSummary` still answers only `texts` and `keys`, with the same content.

## Mutation check

Each mutant was applied, the tests run, and the file restored byte for byte. All 9 were killed:

- a field recorded only when its text isn't already in `texts`;
- a `Text`'s own paragraph counted as a raw `RichText`;
- raw `RichText`s not listed;
- `truncated` at the cap itself (`>=`);
- `truncated` never set;
- counts taken after the cut;
- an obscured field read;
- the RPC answering the old summary;
- no `max`.

## Test plan

- [x] The `probe_agent` suite passes: 298, plus 2 that need the agent build flag. `flutter analyze` is clean.
- [x] `go test ./internal/probelink/ ./internal/runner/` pass; the new keys are ignored.
- [x] Docs: `tools/agent.md` (what the answer holds), `docs/compat/MATRIX.md` (the obscured-field rule covers `editable_texts`), `probe_agent/CHANGELOG.md`, and `CHANGELOG.md` (Added, last).

## The text matcher's sources and the summary

The finder (`_findByText`, `_matchesText`) matches a text selector against these sources. Here is how each one relates to the summary:

| Matcher source | In the summary |
|---|---|
| `Text` with a `data` string (the matcher's `Text` branch reads only `data`) | `texts`, as before |
| a `Text.rich`, which the matcher matches through its inner `RichText` | `texts`: the summary reads the `Text`'s `textSpan`, so its inner `RichText` isn't listed again (#806) |
| a `RichText` that no `Text` built | `rich_texts` (new), except an `Icon`'s own `RichText`, which holds only the icon's glyph (a private-use character, not a text a person reads or a test names). Without that exception, 26 icons would set `truncated`. **An `Icon`'s glyph is still matchable by a text selector** (`findElements` with the glyph's code point finds the `Icon`'s `RichText`), **but it isn't listed in `rich_texts`. A reader must refuse a text selector made of private-use characters.** A test pins that the glyph stays matchable, so this note stays true. |
| a field's value, when the field isn't obscured | `texts` (de-duplicated, as before) and `editable_texts` (new, not de-duplicated against `texts`) |
| an obscured field's value | never matched, and in no key, including `counts`. The test checks the counts with an obscured field present. |
| `InputDecoration` label, hint, helper and error text | these are `Text` widgets inside the field, so they're in `texts` |
| `Tooltip.message` and a `Semantics` label | not listed. The finder uses them **only when no visible text matches**, so for a text that some visible text matches, they're never a second target. A reader that counts a text among `texts`, `editable_texts` and `rich_texts` sees every target the finder would use first. |


## Review follow-up

- **Merged main.** In the CHANGELOG, this PR's bullet is last in Added.
- **Visibility is covered by a test now.** A field and a `RichText` laid out off the screen aren't listed. Offstage and hidden ones aren't either, but those are dropped by the tree walk itself (`_visitElement` skips `Offstage` and `Visibility(visible: false)`), so only the off-screen cases exercise `_isVisible`. My first version of this test used only Offstage content, and both mutants survived it; with the off-screen cases they're killed.
- **An `Icon`'s glyph isn't a rich text.** `_builtByIcon` (the nearest `Icon` above, with no other `RichText` between) leaves it out. Test: 26 icons give an empty `rich_texts` and `truncated: false`. The matcher would match the glyph only for a selector holding that private-use character.
- **Mutants:** 3 more killed (field visibility, `RichText` visibility, the `Icon` exclusion), so 12 of 12 overall.
- **Performance:** the old early exit once both lists were full is gone, so `_isVisible` now runs for every candidate item. The old walk already visited every element; the extra cost is the visibility check per text, field and `RichText`.

- **Icon glyphs (delta review):** the matcher can match an `Icon`'s glyph by a text selector, and `_builtByIcon` leaves it out of every list. So a glyph shown both by an `Icon` and as a real `Text` would be counted once in the summary while the agent matches two. Agreed resolution: the reader refuses text selectors made of private-use characters; that's Studio's side (N8). This PR says so in the table and in `agent.md`. A test pins that `findElements` still matches the glyph.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

see exactly N counts an ordinary Text twice ("found 2" for one Text)

1 participant