Skip to content

feat(fixtures): Text and display (text) widget family, measured on two named devices - #794

Merged
pbertsch merged 3 commits into
mainfrom
test/compat-text-display
Nov 17, 2026
Merged

pbertsch merged 3 commits into
mainfrom
test/compat-text-display

Conversation

@pbertsch

@pbertsch pbertsch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Widget family: Text and display (the text part). Item 1 of docs/compat/NEXT.md: a fixture screen, a ProbeScript suite, the variants that must fail, and the matrix results, set only from runs on named devices.

  • Screen (PROBE_FIXTURE_WIDGETS=display, native-test-apps/flutter/lib/widget_families.dart):

    • Text("Order total: 42.00") (wf_text);
    • a one-line Text cut with an ellipsis, longer than the screen is wide (wf_ellipsis);
    • a Text.rich of "Hello " and a bold "world" (wf_rich);
    • a RichText of "Painted " and "rich text" (wf_richtext).

    Each has a Semantics(identifier:) and a ValueKey of the same name.

  • Suite probe-tests/widgets/display.probe (5 tests):

    • a plain text by its whole text, a part, and contains;
    • the spans of Text.rich, and of RichText, as one text and as parts;
    • the cut text by its whole string, including the part that isn't painted;
    • every widget by #id.
  • Negative variants probe-tests/negative/display.probe.expect-fail (6 tests): a number one cent off; contains a value the text doesn't hold; a text that isn't on the screen; another case; two spaces between the spans; the painted, cut form.

  • Matrix: a "Text and display: text" section, and the family row points to it. Cells are set for Text and RichText only; Icon, Image and the rest are item 3. NEXT.md item 1 is marked assessed; the README lists the family. CHANGELOG under Added.

What it settles

  • A text selector matches the text the app gave the widget, not what is painted. The cut text is found by its whole string, including "United States of America", which is not on screen, and not by the cut form with ….
  • It matches the whole text or any part of it, exactly. Case counts and spaces count. The spans of a Text.rich or a RichText are matched as one text, joined as written.
  • Found, and filed as Steps right after restart the app can run before the first frame has settled on slow devices #793: right after restart the app, the agent can answer while the screen is still being built. On Compat-Pixel-API35, the suite without a wait failed 3 of 10 runs; with a wait for the first item, 1 of 10; with a wait for the screen's last widget, 10 of 10 passed. The suite waits for the last widget, with a comment pointing to Steps right after restart the app can run before the first frame has settled on slow devices #793. The merged buttons suite fails 7 of 10 runs there for the same reason; that's for a separate PR.

Evidence (devices)

Flutter 3.44.8 (installed; pubspec.lock unchanged). The probe CLI was built from this branch; the agent is the fixture's path dependency (0.27.0).

  • Android: Compat-Pixel-API35, Android 15 (API 35), emulator-5590, name read back before each step; host port 48690, device port 48686.
  • iOS: Compat-iPhone, UDID A57AEE67-3473-4C20-BE25-8847C2A9D6BE, iPhone 17 Pro, iOS 26.5, by UDID; the agent is built with PROBE_PORT=48691.
Run (final suite, with the wait) Compat-Pixel-API35 Compat-iPhone
display.probe 5 of 5 tests, in 10 of 10 runs 5 of 5 tests, in 5 of 5 runs (and 3 of 3 before the wait was added)
the negative file 0 passed, 6 failed, each with its stated message 0 passed, 6 failed, each with its stated message

Before writing the suite, two exploratory files (14 and 7 questions) were run on both devices with the same results; they are not committed.

Flutter 3.50.0 was not run for this family; the matrix says so.

Not covered

SelectableText; text scale; right-to-left and other scripts; semanticsLabel (item 3); soft wrapping without maxLines; Icon, Image, FlutterLogo, CircleAvatar (item 3); a physical device.

Test plan

  • The suite and the negative file on both named devices (table above).
  • make ci-local: 5 of 5 required checks at 90a7d2e.
  • go test ./scripts/: the negative-suite guards accept the new file.
  • dart format and flutter analyze lib: no issues.

Review follow-up

  • Read tests for every read cell. These are now in the suite, each with a negative variant:

    • see "Hello world" contains "world" (N7 contains "earth");
    • see "Painted rich text" contains "rich text" (N8 contains "plain text");
    • see "Shipping address" contains "United States of America" (N9 contains "Canada").

    What contains reads for a Text.rich: the failing N7 shows the text read is the whole "Hello world", not an empty string. The matrix read cells say this.

  • The ellipsis text is described as chosen longer than the screen is wide: an inference from its length, since the agent doesn't report the painted width.

  • N6 says it shows the finding by its construction.

  • The README comma after "Cupertino fields)" is fixed.

  • NEXT.md item 1 says "find and read", and that Icon and Image stay item 3.

  • The 3 of 10 to 10 of 10 figure stays labelled as measured on Compat-Pixel-API35.

  • Main merged in, after the navigation family: both families are in the fixture, README, matrix, NEXT and CHANGELOG.

  • Rerun after the change: display.probe passed 5 of 5 tests in 10 of 10 runs on Compat-Pixel-API35 and in 5 of 5 runs on Compat-iPhone. The negative file, now 9 tests, gave 0 passed and 9 failed on each device, each with its stated message. One more Android run after the merge passed 5 of 5.

@pbertsch
pbertsch requested a review from a team as a code owner November 17, 2026 10:58
@pbertsch
pbertsch merged commit c7d53df into main Nov 17, 2026
10 of 12 checks passed
pbertsch added a commit that referenced this pull request Nov 17, 2026
…app (issue #793) (#795)

## Summary

Interim fix for #793: the widget-family suites of the Flutter fixture wait for their screen's last element right after `restart the app`, so they no longer fail on a slower device.

After a relaunch, an app can draw its first frames with a 0×0 view for about a second (measured on an Android 15 emulator; see the issue). A `ListView` laid out then builds only its first child, and a step that runs in that window sees half the screen. Each suite's `before each test` now does:

```
restart the app
# the last element of the screen: right after a restart the agent can answer while the screen is still being built (issue #793)
wait until #<last element> appears
```

Last elements: `buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`. The negative files of `buttons` and `navigation` get the same wait, so they fail for the reason they state and not because of the restart.

This changes only the suites. The fix for the cause is in the agent: it should answer only once the view has its real size. That comes in a separate PR, after which these waits can stay as harmless or be removed.

## Evidence

AVD `Compat-Pixel-API35`, Android 15 (API 35), `emulator-5590`, name read back before each run; Flutter 3.44.8. Each suite ran 10 times, as on `main` and as in this branch:

| Suite | `main` | this branch |
|---|---|---|
| `buttons.probe` | 3 of 10 runs passed | 10 of 10 |
| `selection.probe` | 7 of 10 | 10 of 10 |
| `textinput.probe` | 3 of 10 | 10 of 10 |
| `navigation.probe` | 10 of 10 | 10 of 10 |

The negative files, run once each with the wait: `buttons` 0 passed, 8 failed; `navigation` 0 passed, 5 failed; each failure gives its stated message.

Not run: the iOS simulator. The suites passed there in every earlier run without the wait.

## Test plan

- [x] The table above.
- [x] `go test ./scripts/`: the negative-suite guards accept the changed files.

## Review follow-up

- **The comment on each wait** says the target must be on screen: `wait until ... appears` matches only what is on screen, so the wait needs a screen tall enough to show the screen's last element with an id (measured on a Pixel 6 profile), and on a shorter screen it would time out. The fix in the agent removes the need.
- **An earlier target was tried and measured, and was not enough.** I tried the list's second element with an id (`#wf_elevated_state`, `#wf_checkbox_state`, `#wf_textfield_state`), which is on screen on any phone. On `Compat-Pixel-API35`: `buttons` 10 of 10, `selection` 10 of 10, but `textinput` **8 of 10**. One failure listed the visible texts as only the first two items (`"Plain text"`, `"Plain: (empty)"`): the screen can be partly built past the first item. So the waits stay on the last element (`buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`), which measured 10 of 10 on all four (the table above).
- **Negative files.** Each header says a failure at the shared `wait until` (a timeout) is not the expected failure. I read every failure message after adding the wait; each test fails at its own step:
  - `buttons` (0 passed, 8 failed):
    - N1 `see "Elevated: 2"` not found;
    - N2 `"Press elevated"` is enabled;
    - N3 `"Cannot press"` is disabled;
    - N4 `see "Disabled: 1"` not found;
    - N5 `see "Elevated: 0"` not found;
    - N6 `tap "Press elevatd"`: widget not found, with the full list of visible texts;
    - N7 `see "Hold: tap 0, long 1"` not found;
    - N8 `see "Icon button: 1"` not found.
  - `navigation` (0 passed, 5 failed):
    - N1 `see "Inbox"` not found;
    - N2 `see "Queries: 2"` not found;
    - N3 `see "From: Ada"` not found;
    - N4 `see "From: Ada"` not found;
    - N5 `tap "Serch"`: widget not found.

  None failed at the wait.
- Main is merged in, after #794.
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…o named devices (AlphaWaveSystems#794)

## Summary

**Widget family: Text and display (the text part).** Item 1 of `docs/compat/NEXT.md`: a fixture screen, a ProbeScript suite, the variants that must fail, and the matrix results, set only from runs on named devices.

- **Screen** (`PROBE_FIXTURE_WIDGETS=display`, `native-test-apps/flutter/lib/widget_families.dart`):
  - `Text("Order total: 42.00")` (`wf_text`);
  - a one-line `Text` cut with an ellipsis, longer than the screen is wide (`wf_ellipsis`);
  - a `Text.rich` of "Hello " and a bold "world" (`wf_rich`);
  - a `RichText` of "Painted " and "rich text" (`wf_richtext`).

  Each has a `Semantics(identifier:)` and a `ValueKey` of the same name.
- **Suite** `probe-tests/widgets/display.probe` (5 tests):
  - a plain text by its whole text, a part, and `contains`;
  - the spans of `Text.rich`, and of `RichText`, as one text and as parts;
  - the cut text by its whole string, including the part that isn't painted;
  - every widget by `#id`.
- **Negative variants** `probe-tests/negative/display.probe.expect-fail` (6 tests): a number one cent off; `contains` a value the text doesn't hold; a text that isn't on the screen; another case; two spaces between the spans; the painted, cut form.
- **Matrix:** a "Text and display: text" section, and the family row points to it. Cells are set for `Text` and `RichText` only; `Icon`, `Image` and the rest are item 3. `NEXT.md` item 1 is marked assessed; the README lists the family. CHANGELOG under Added.

## What it settles

- **A text selector matches the text the app gave the widget, not what is painted.** The cut text is found by its whole string, including "United States of America", which is not on screen, and not by the cut form with `…`.
- **It matches the whole text or any part of it, exactly.** Case counts and spaces count. The spans of a `Text.rich` or a `RichText` are matched as one text, joined as written.
- **Found, and filed as AlphaWaveSystems#793:** right after `restart the app`, the agent can answer while the screen is still being built. On `Compat-Pixel-API35`, the suite without a wait failed 3 of 10 runs; with a wait for the first item, 1 of 10; with a wait for the screen's last widget, 10 of 10 passed. The suite waits for the last widget, with a comment pointing to AlphaWaveSystems#793. The merged `buttons` suite fails 7 of 10 runs there for the same reason; that's for a separate PR.

## Evidence (devices)

Flutter **3.44.8** (installed; `pubspec.lock` unchanged). The probe CLI was built from this branch; the agent is the fixture's path dependency (0.27.0).

- **Android:** `Compat-Pixel-API35`, Android 15 (API 35), `emulator-5590`, name read back before each step; host port 48690, device port 48686.
- **iOS:** `Compat-iPhone`, UDID `A57AEE67-3473-4C20-BE25-8847C2A9D6BE`, iPhone 17 Pro, iOS 26.5, by UDID; the agent is built with `PROBE_PORT=48691`.

| Run (final suite, with the wait) | `Compat-Pixel-API35` | `Compat-iPhone` |
|---|---|---|
| `display.probe` | 5 of 5 tests, in 10 of 10 runs | 5 of 5 tests, in 5 of 5 runs (and 3 of 3 before the wait was added) |
| the negative file | 0 passed, 6 failed, each with its stated message | 0 passed, 6 failed, each with its stated message |

Before writing the suite, two exploratory files (14 and 7 questions) were run on both devices with the same results; they are not committed.

Flutter 3.50.0 was not run for this family; the matrix says so.

## Not covered

`SelectableText`; text scale; right-to-left and other scripts; `semanticsLabel` (item 3); soft wrapping without `maxLines`; `Icon`, `Image`, `FlutterLogo`, `CircleAvatar` (item 3); a physical device.

## Test plan

- [x] The suite and the negative file on both named devices (table above).
- [x] `make ci-local`: 5 of 5 required checks at 90a7d2e.
- [x] `go test ./scripts/`: the negative-suite guards accept the new file.
- [x] `dart format` and `flutter analyze lib`: no issues.

## Review follow-up

- **Read tests for every read cell.** These are now in the suite, each with a negative variant:
  - `see "Hello world" contains "world"` (N7 `contains "earth"`);
  - `see "Painted rich text" contains "rich text"` (N8 `contains "plain text"`);
  - `see "Shipping address" contains "United States of America"` (N9 `contains "Canada"`).

  What `contains` reads for a `Text.rich`: the failing N7 shows the text read is the whole `"Hello world"`, not an empty string. The matrix read cells say this.
- **The ellipsis text** is described as *chosen* longer than the screen is wide: an inference from its length, since the agent doesn't report the painted width.
- **N6** says it shows the finding by its construction.
- **The README comma** after "Cupertino fields)" is fixed.
- **`NEXT.md` item 1** says "find and read", and that `Icon` and `Image` stay item 3.
- The 3 of 10 to 10 of 10 figure stays labelled as measured on `Compat-Pixel-API35`.
- **Main merged in**, after the navigation family: both families are in the fixture, README, matrix, NEXT and CHANGELOG.
- **Rerun after the change:** `display.probe` passed 5 of 5 tests in 10 of 10 runs on `Compat-Pixel-API35` and in 5 of 5 runs on `Compat-iPhone`. The negative file, now 9 tests, gave 0 passed and 9 failed on each device, each with its stated message. One more Android run after the merge passed 5 of 5.
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…app (issue AlphaWaveSystems#793) (AlphaWaveSystems#795)

## Summary

Interim fix for AlphaWaveSystems#793: the widget-family suites of the Flutter fixture wait for their screen's last element right after `restart the app`, so they no longer fail on a slower device.

After a relaunch, an app can draw its first frames with a 0×0 view for about a second (measured on an Android 15 emulator; see the issue). A `ListView` laid out then builds only its first child, and a step that runs in that window sees half the screen. Each suite's `before each test` now does:

```
restart the app
# the last element of the screen: right after a restart the agent can answer while the screen is still being built (issue AlphaWaveSystems#793)
wait until #<last element> appears
```

Last elements: `buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`. The negative files of `buttons` and `navigation` get the same wait, so they fail for the reason they state and not because of the restart.

This changes only the suites. The fix for the cause is in the agent: it should answer only once the view has its real size. That comes in a separate PR, after which these waits can stay as harmless or be removed.

## Evidence

AVD `Compat-Pixel-API35`, Android 15 (API 35), `emulator-5590`, name read back before each run; Flutter 3.44.8. Each suite ran 10 times, as on `main` and as in this branch:

| Suite | `main` | this branch |
|---|---|---|
| `buttons.probe` | 3 of 10 runs passed | 10 of 10 |
| `selection.probe` | 7 of 10 | 10 of 10 |
| `textinput.probe` | 3 of 10 | 10 of 10 |
| `navigation.probe` | 10 of 10 | 10 of 10 |

The negative files, run once each with the wait: `buttons` 0 passed, 8 failed; `navigation` 0 passed, 5 failed; each failure gives its stated message.

Not run: the iOS simulator. The suites passed there in every earlier run without the wait.

## Test plan

- [x] The table above.
- [x] `go test ./scripts/`: the negative-suite guards accept the changed files.

## Review follow-up

- **The comment on each wait** says the target must be on screen: `wait until ... appears` matches only what is on screen, so the wait needs a screen tall enough to show the screen's last element with an id (measured on a Pixel 6 profile), and on a shorter screen it would time out. The fix in the agent removes the need.
- **An earlier target was tried and measured, and was not enough.** I tried the list's second element with an id (`#wf_elevated_state`, `#wf_checkbox_state`, `#wf_textfield_state`), which is on screen on any phone. On `Compat-Pixel-API35`: `buttons` 10 of 10, `selection` 10 of 10, but `textinput` **8 of 10**. One failure listed the visible texts as only the first two items (`"Plain text"`, `"Plain: (empty)"`): the screen can be partly built past the first item. So the waits stay on the last element (`buttons` `#wf_long_press_state`, `selection` `#wf_cupertino_switch_state`, `textinput` `#wf_cupertino_state`, `navigation` `#wf_open_message`), which measured 10 of 10 on all four (the table above).
- **Negative files.** Each header says a failure at the shared `wait until` (a timeout) is not the expected failure. I read every failure message after adding the wait; each test fails at its own step:
  - `buttons` (0 passed, 8 failed):
    - N1 `see "Elevated: 2"` not found;
    - N2 `"Press elevated"` is enabled;
    - N3 `"Cannot press"` is disabled;
    - N4 `see "Disabled: 1"` not found;
    - N5 `see "Elevated: 0"` not found;
    - N6 `tap "Press elevatd"`: widget not found, with the full list of visible texts;
    - N7 `see "Hold: tap 0, long 1"` not found;
    - N8 `see "Icon button: 1"` not found.
  - `navigation` (0 passed, 5 failed):
    - N1 `see "Inbox"` not found;
    - N2 `see "Queries: 2"` not found;
    - N3 `see "From: Ada"` not found;
    - N4 `see "From: Ada"` not found;
    - N5 `tap "Serch"`: widget not found.

  None failed at the wait.
- Main is merged in, after AlphaWaveSystems#794.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant