Repository navigation
feat(fixtures): Text and display (text) widget family, measured on two named devices - #794
Merged
Merged
Conversation
… negative variants and matrix results
2 tasks done
pbertsch
added a commit
that referenced
this pull request
Nov 17, 2026
…app (issue #793) (#795) ## Summary Interim fix for #793: the widget-family suites of the Flutter fixture wait for their screen's last element right after `restart the app`, so they no longer fail on a slower device. After a relaunch, an app can draw its first frames with a 0×0 view for about a second (measured on an Android 15 emulator; see the issue). A `ListView` laid out then builds only its first child, and a step that runs in that window sees half the screen. Each suite's `before each test` now does: ``` restart the app # the last element of the screen: right after a restart the agent can answer while the screen is still being built (issue #793) wait until #<last element> appears ``` Last elements: `buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`. The negative files of `buttons` and `navigation` get the same wait, so they fail for the reason they state and not because of the restart. This changes only the suites. The fix for the cause is in the agent: it should answer only once the view has its real size. That comes in a separate PR, after which these waits can stay as harmless or be removed. ## Evidence AVD `Compat-Pixel-API35`, Android 15 (API 35), `emulator-5590`, name read back before each run; Flutter 3.44.8. Each suite ran 10 times, as on `main` and as in this branch: | Suite | `main` | this branch | |---|---|---| | `buttons.probe` | 3 of 10 runs passed | 10 of 10 | | `selection.probe` | 7 of 10 | 10 of 10 | | `textinput.probe` | 3 of 10 | 10 of 10 | | `navigation.probe` | 10 of 10 | 10 of 10 | The negative files, run once each with the wait: `buttons` 0 passed, 8 failed; `navigation` 0 passed, 5 failed; each failure gives its stated message. Not run: the iOS simulator. The suites passed there in every earlier run without the wait. ## Test plan - [x] The table above. - [x] `go test ./scripts/`: the negative-suite guards accept the changed files. ## Review follow-up - **The comment on each wait** says the target must be on screen: `wait until ... appears` matches only what is on screen, so the wait needs a screen tall enough to show the screen's last element with an id (measured on a Pixel 6 profile), and on a shorter screen it would time out. The fix in the agent removes the need. - **An earlier target was tried and measured, and was not enough.** I tried the list's second element with an id (`#wf_elevated_state`, `#wf_checkbox_state`, `#wf_textfield_state`), which is on screen on any phone. On `Compat-Pixel-API35`: `buttons` 10 of 10, `selection` 10 of 10, but `textinput` **8 of 10**. One failure listed the visible texts as only the first two items (`"Plain text"`, `"Plain: (empty)"`): the screen can be partly built past the first item. So the waits stay on the last element (`buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`), which measured 10 of 10 on all four (the table above). - **Negative files.** Each header says a failure at the shared `wait until` (a timeout) is not the expected failure. I read every failure message after adding the wait; each test fails at its own step: - `buttons` (0 passed, 8 failed): - N1 `see "Elevated: 2"` not found; - N2 `"Press elevated"` is enabled; - N3 `"Cannot press"` is disabled; - N4 `see "Disabled: 1"` not found; - N5 `see "Elevated: 0"` not found; - N6 `tap "Press elevatd"`: widget not found, with the full list of visible texts; - N7 `see "Hold: tap 0, long 1"` not found; - N8 `see "Icon button: 1"` not found. - `navigation` (0 passed, 5 failed): - N1 `see "Inbox"` not found; - N2 `see "Queries: 2"` not found; - N3 `see "From: Ada"` not found; - N4 `see "From: Ada"` not found; - N5 `tap "Serch"`: widget not found. None failed at the wait. - Main is merged in, after #794.
jay23606
added a commit
to jay23606/flutter-probe
that referenced
this pull request
Nov 17, 2026
…o named devices (AlphaWaveSystems#794) ## Summary **Widget family: Text and display (the text part).** Item 1 of `docs/compat/NEXT.md`: a fixture screen, a ProbeScript suite, the variants that must fail, and the matrix results, set only from runs on named devices. - **Screen** (`PROBE_FIXTURE_WIDGETS=display`, `native-test-apps/flutter/lib/widget_families.dart`): - `Text("Order total: 42.00")` (`wf_text`); - a one-line `Text` cut with an ellipsis, longer than the screen is wide (`wf_ellipsis`); - a `Text.rich` of "Hello " and a bold "world" (`wf_rich`); - a `RichText` of "Painted " and "rich text" (`wf_richtext`). Each has a `Semantics(identifier:)` and a `ValueKey` of the same name. - **Suite** `probe-tests/widgets/display.probe` (5 tests): - a plain text by its whole text, a part, and `contains`; - the spans of `Text.rich`, and of `RichText`, as one text and as parts; - the cut text by its whole string, including the part that isn't painted; - every widget by `#id`. - **Negative variants** `probe-tests/negative/display.probe.expect-fail` (6 tests): a number one cent off; `contains` a value the text doesn't hold; a text that isn't on the screen; another case; two spaces between the spans; the painted, cut form. - **Matrix:** a "Text and display: text" section, and the family row points to it. Cells are set for `Text` and `RichText` only; `Icon`, `Image` and the rest are item 3. `NEXT.md` item 1 is marked assessed; the README lists the family. CHANGELOG under Added. ## What it settles - **A text selector matches the text the app gave the widget, not what is painted.** The cut text is found by its whole string, including "United States of America", which is not on screen, and not by the cut form with `…`. - **It matches the whole text or any part of it, exactly.** Case counts and spaces count. The spans of a `Text.rich` or a `RichText` are matched as one text, joined as written. - **Found, and filed as AlphaWaveSystems#793:** right after `restart the app`, the agent can answer while the screen is still being built. On `Compat-Pixel-API35`, the suite without a wait failed 3 of 10 runs; with a wait for the first item, 1 of 10; with a wait for the screen's last widget, 10 of 10 passed. The suite waits for the last widget, with a comment pointing to AlphaWaveSystems#793. The merged `buttons` suite fails 7 of 10 runs there for the same reason; that's for a separate PR. ## Evidence (devices) Flutter **3.44.8** (installed; `pubspec.lock` unchanged). The probe CLI was built from this branch; the agent is the fixture's path dependency (0.27.0). - **Android:** `Compat-Pixel-API35`, Android 15 (API 35), `emulator-5590`, name read back before each step; host port 48690, device port 48686. - **iOS:** `Compat-iPhone`, UDID `A57AEE67-3473-4C20-BE25-8847C2A9D6BE`, iPhone 17 Pro, iOS 26.5, by UDID; the agent is built with `PROBE_PORT=48691`. | Run (final suite, with the wait) | `Compat-Pixel-API35` | `Compat-iPhone` | |---|---|---| | `display.probe` | 5 of 5 tests, in 10 of 10 runs | 5 of 5 tests, in 5 of 5 runs (and 3 of 3 before the wait was added) | | the negative file | 0 passed, 6 failed, each with its stated message | 0 passed, 6 failed, each with its stated message | Before writing the suite, two exploratory files (14 and 7 questions) were run on both devices with the same results; they are not committed. Flutter 3.50.0 was not run for this family; the matrix says so. ## Not covered `SelectableText`; text scale; right-to-left and other scripts; `semanticsLabel` (item 3); soft wrapping without `maxLines`; `Icon`, `Image`, `FlutterLogo`, `CircleAvatar` (item 3); a physical device. ## Test plan - [x] The suite and the negative file on both named devices (table above). - [x] `make ci-local`: 5 of 5 required checks at 90a7d2e. - [x] `go test ./scripts/`: the negative-suite guards accept the new file. - [x] `dart format` and `flutter analyze lib`: no issues. ## Review follow-up - **Read tests for every read cell.** These are now in the suite, each with a negative variant: - `see "Hello world" contains "world"` (N7 `contains "earth"`); - `see "Painted rich text" contains "rich text"` (N8 `contains "plain text"`); - `see "Shipping address" contains "United States of America"` (N9 `contains "Canada"`). What `contains` reads for a `Text.rich`: the failing N7 shows the text read is the whole `"Hello world"`, not an empty string. The matrix read cells say this. - **The ellipsis text** is described as *chosen* longer than the screen is wide: an inference from its length, since the agent doesn't report the painted width. - **N6** says it shows the finding by its construction. - **The README comma** after "Cupertino fields)" is fixed. - **`NEXT.md` item 1** says "find and read", and that `Icon` and `Image` stay item 3. - The 3 of 10 to 10 of 10 figure stays labelled as measured on `Compat-Pixel-API35`. - **Main merged in**, after the navigation family: both families are in the fixture, README, matrix, NEXT and CHANGELOG. - **Rerun after the change:** `display.probe` passed 5 of 5 tests in 10 of 10 runs on `Compat-Pixel-API35` and in 5 of 5 runs on `Compat-iPhone`. The negative file, now 9 tests, gave 0 passed and 9 failed on each device, each with its stated message. One more Android run after the merge passed 5 of 5.
jay23606
added a commit
to jay23606/flutter-probe
that referenced
this pull request
Nov 17, 2026
…app (issue AlphaWaveSystems#793) (AlphaWaveSystems#795) ## Summary Interim fix for AlphaWaveSystems#793: the widget-family suites of the Flutter fixture wait for their screen's last element right after `restart the app`, so they no longer fail on a slower device. After a relaunch, an app can draw its first frames with a 0×0 view for about a second (measured on an Android 15 emulator; see the issue). A `ListView` laid out then builds only its first child, and a step that runs in that window sees half the screen. Each suite's `before each test` now does: ``` restart the app # the last element of the screen: right after a restart the agent can answer while the screen is still being built (issue AlphaWaveSystems#793) wait until #<last element> appears ``` Last elements: `buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`. The negative files of `buttons` and `navigation` get the same wait, so they fail for the reason they state and not because of the restart. This changes only the suites. The fix for the cause is in the agent: it should answer only once the view has its real size. That comes in a separate PR, after which these waits can stay as harmless or be removed. ## Evidence AVD `Compat-Pixel-API35`, Android 15 (API 35), `emulator-5590`, name read back before each run; Flutter 3.44.8. Each suite ran 10 times, as on `main` and as in this branch: | Suite | `main` | this branch | |---|---|---| | `buttons.probe` | 3 of 10 runs passed | 10 of 10 | | `selection.probe` | 7 of 10 | 10 of 10 | | `textinput.probe` | 3 of 10 | 10 of 10 | | `navigation.probe` | 10 of 10 | 10 of 10 | The negative files, run once each with the wait: `buttons` 0 passed, 8 failed; `navigation` 0 passed, 5 failed; each failure gives its stated message. Not run: the iOS simulator. The suites passed there in every earlier run without the wait. ## Test plan - [x] The table above. - [x] `go test ./scripts/`: the negative-suite guards accept the changed files. ## Review follow-up - **The comment on each wait** says the target must be on screen: `wait until ... appears` matches only what is on screen, so the wait needs a screen tall enough to show the screen's last element with an id (measured on a Pixel 6 profile), and on a shorter screen it would time out. The fix in the agent removes the need. - **An earlier target was tried and measured, and was not enough.** I tried the list's second element with an id (`#wf_elevated_state`, `#wf_checkbox_state`, `#wf_textfield_state`), which is on screen on any phone. On `Compat-Pixel-API35`: `buttons` 10 of 10, `selection` 10 of 10, but `textinput` **8 of 10**. One failure listed the visible texts as only the first two items (`"Plain text"`, `"Plain: (empty)"`): the screen can be partly built past the first item. So the waits stay on the last element (`buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`), which measured 10 of 10 on all four (the table above). - **Negative files.** Each header says a failure at the shared `wait until` (a timeout) is not the expected failure. I read every failure message after adding the wait; each test fails at its own step: - `buttons` (0 passed, 8 failed): - N1 `see "Elevated: 2"` not found; - N2 `"Press elevated"` is enabled; - N3 `"Cannot press"` is disabled; - N4 `see "Disabled: 1"` not found; - N5 `see "Elevated: 0"` not found; - N6 `tap "Press elevatd"`: widget not found, with the full list of visible texts; - N7 `see "Hold: tap 0, long 1"` not found; - N8 `see "Icon button: 1"` not found. - `navigation` (0 passed, 5 failed): - N1 `see "Inbox"` not found; - N2 `see "Queries: 2"` not found; - N3 `see "From: Ada"` not found; - N4 `see "From: Ada"` not found; - N5 `tap "Serch"`: widget not found. None failed at the wait. - Main is merged in, after AlphaWaveSystems#794.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Widget family: Text and display (the text part). Item 1 of
docs/compat/NEXT.md: a fixture screen, a ProbeScript suite, the variants that must fail, and the matrix results, set only from runs on named devices.Screen (
PROBE_FIXTURE_WIDGETS=display,native-test-apps/flutter/lib/widget_families.dart):Text("Order total: 42.00")(wf_text);Textcut with an ellipsis, longer than the screen is wide (wf_ellipsis);Text.richof "Hello " and a bold "world" (wf_rich);RichTextof "Painted " and "rich text" (wf_richtext).Each has a
Semantics(identifier:)and aValueKeyof the same name.Suite
probe-tests/widgets/display.probe(5 tests):contains;Text.rich, and ofRichText, as one text and as parts;#id.Negative variants
probe-tests/negative/display.probe.expect-fail(6 tests): a number one cent off;containsa value the text doesn't hold; a text that isn't on the screen; another case; two spaces between the spans; the painted, cut form.Matrix: a "Text and display: text" section, and the family row points to it. Cells are set for
TextandRichTextonly;Icon,Imageand the rest are item 3.NEXT.mditem 1 is marked assessed; the README lists the family. CHANGELOG under Added.What it settles
….Text.richor aRichTextare matched as one text, joined as written.restart the appcan run before the first frame has settled on slow devices #793: right afterrestart the app, the agent can answer while the screen is still being built. OnCompat-Pixel-API35, the suite without a wait failed 3 of 10 runs; with a wait for the first item, 1 of 10; with a wait for the screen's last widget, 10 of 10 passed. The suite waits for the last widget, with a comment pointing to Steps right afterrestart the appcan run before the first frame has settled on slow devices #793. The mergedbuttonssuite fails 7 of 10 runs there for the same reason; that's for a separate PR.Evidence (devices)
Flutter 3.44.8 (installed;
pubspec.lockunchanged). The probe CLI was built from this branch; the agent is the fixture's path dependency (0.27.0).Compat-Pixel-API35, Android 15 (API 35),emulator-5590, name read back before each step; host port 48690, device port 48686.Compat-iPhone, UDIDA57AEE67-3473-4C20-BE25-8847C2A9D6BE, iPhone 17 Pro, iOS 26.5, by UDID; the agent is built withPROBE_PORT=48691.Compat-Pixel-API35Compat-iPhonedisplay.probeBefore writing the suite, two exploratory files (14 and 7 questions) were run on both devices with the same results; they are not committed.
Flutter 3.50.0 was not run for this family; the matrix says so.
Not covered
SelectableText; text scale; right-to-left and other scripts;semanticsLabel(item 3); soft wrapping withoutmaxLines;Icon,Image,FlutterLogo,CircleAvatar(item 3); a physical device.Test plan
make ci-local: 5 of 5 required checks at 90a7d2e.go test ./scripts/: the negative-suite guards accept the new file.dart formatandflutter analyze lib: no issues.Review follow-up
Read tests for every read cell. These are now in the suite, each with a negative variant:
see "Hello world" contains "world"(N7contains "earth");see "Painted rich text" contains "rich text"(N8contains "plain text");see "Shipping address" contains "United States of America"(N9contains "Canada").What
containsreads for aText.rich: the failing N7 shows the text read is the whole"Hello world", not an empty string. The matrix read cells say this.The ellipsis text is described as chosen longer than the screen is wide: an inference from its length, since the agent doesn't report the painted width.
N6 says it shows the finding by its construction.
The README comma after "Cupertino fields)" is fixed.
NEXT.mditem 1 says "find and read", and thatIconandImagestay item 3.The 3 of 10 to 10 of 10 figure stays labelled as measured on
Compat-Pixel-API35.Main merged in, after the navigation family: both families are in the fixture, README, matrix, NEXT and CHANGELOG.
Rerun after the change:
display.probepassed 5 of 5 tests in 10 of 10 runs onCompat-Pixel-API35and in 5 of 5 runs onCompat-iPhone. The negative file, now 9 tests, gave 0 passed and 9 failed on each device, each with its stated message. One more Android run after the merge passed 5 of 5.