Skip to content

feat(fixtures): Navigation widget family: app bar and routes, measured on two named devices - #792

Merged
pbertsch merged 3 commits into
mainfrom
test/compat-route-under-top
Nov 17, 2026
Merged

pbertsch merged 3 commits into
mainfrom
test/compat-route-under-top

Conversation

@pbertsch

@pbertsch pbertsch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Widget family: Navigation (app bar and routes). Item 2 of docs/compat/NEXT.md: a fixture screen, a ProbeScript suite, the variants that must fail, and the matrix results, set only from runs on named devices.

  • Screen (PROBE_FIXTURE_WIDGETS=navigation, native-test-apps/flutter/lib/widget_families.dart):

    • a Scaffold whose AppBar is titled "Inbox", with an action IconButton with the tooltip "Search" (wf_search), counted in the state text Queries: N;
    • a button "Open message" (wf_open_message) that pushes a MaterialPageRoute titled "Message", whose body is From: Ada.

    No label is part of a state text, since text selectors match substrings; that is why the counter is Queries and not Searches. Every control has a Semantics(identifier:) and a ValueKey of the same name, as in the other families.

  • Suite probe-tests/widgets/navigation.probe (4 tests): the bar's title, and its action by tooltip and by #id; a pushed route is seen; the route under the top one is not seen; go back returns to the first route.

  • Negative variants probe-tests/negative/navigation.probe.expect-fail (5 tests, each with its # Expected: line): the route under the top one seen; one tap counted twice; the second route before it is pushed; the second route after go back; a tooltip that is not on the screen.

  • Matrix: a Navigation section, and the family row now points to it. The cells are set only for AppBar and Navigator; NavigationBar, TabBar and Drawer stay not assessed. NEXT.md item 2 says it is assessed; the README lists the family. CHANGELOG under Added.

What it settles

  • The route rule holds on a device. While the message route (opaque) is on top, the first route's title, texts and button are not matched: don't see "Inbox", "Queries: 0" and "Open message" pass, and see "Inbox" fails. This is the agent's rule (_isVisible and _routeOnScreen in probe_agent/lib/src/finder.dart), which until now had only a unit test (PT-03).
  • Finding: during a pop, the leaving route is still matched, on both devices. Right after go back, wait until "Inbox" appears passes at once, while the message route is still on screen in Material's page transition. The first run of the suite failed at don't see "From: Ada": it found 2 elements. Those steps alone, 5 runs on each device: the leaving route was found in 4 of 5 runs on Compat-Pixel-API35 and in 5 of 5 on Compat-iPhone. A test that checks a pop now waits for the leaving route to go (wait until "From: Ada" disappears). The suite says this in a comment, and the negative N4 uses the same wait.
  • go back is the agent calling Navigator.pop(). It is not the Android back key or the iOS edge swipe; neither was run.

Evidence (devices)

Flutter 3.44.8 (installed; pubspec.lock unchanged, nothing added or upgraded). The probe CLI was built from this branch; the agent is the fixture's path dependency (0.27.0).

  • Android: AVD Compat-Pixel-API35 (created for this work from the installed android-35/google_apis/arm64-v8a image, Pixel 6 profile), Android 15 (API 35), emulator-5590. The name was read back (adb -s emulator-5590 emu avd name) before each step. Host port 48690, device port 48686.
  • iOS: simulator Compat-iPhone, UDID A57AEE67-3473-4C20-BE25-8847C2A9D6BE (created for this work), iPhone 17 Pro, iOS 26.5, always addressed by UDID. The agent was built with --dart-define=PROBE_PORT=48691, its own port, since a simulator shares the host's network.
Run Compat-Pixel-API35 Compat-iPhone
navigation.probe, 3 runs 4 passed, 3 times 4 passed, 3 times
the negative file, run as its header says 0 passed, 5 failed, each with its stated message 0 passed, 5 failed, each with its stated message

Flutter 3.50.0 was not run for this family; the matrix says so. I've asked where a 3.50.0 SDK copy is; if one is available, the suite runs on it in a follow-up and the cells name both versions.

Not covered

NavigationBar, TabBar, Drawer; a route that is not opaque (a dialog, a popup menu, a sheet, where the page under it stays matched by design; item 8); named routes and go_router; the system back key and the iOS edge swipe; a nested Navigator; Cupertino routes (item 10); a physical device; other screen sizes, text scales and accessibility services. CI does not build the fixture.

Test plan

  • The suite and the negative file on both named devices (table above).
  • make ci-local: 5 of 5 required checks at cc5bf40 (Build, Go unit tests, probe-convert tests, Lint, Disclosure guard).
  • go test ./scripts/: the negative-suite guards accept the new file.
  • dart format and flutter analyze lib: no issues.

Review follow-up

  • The transition finding names its devices, in the matrix, the CHANGELOG and here: Android 4 of 5 runs, iOS 5 of 5 (pop-repro: tap, wait for the message, go back, wait until "Inbox" appears, don't see "From: Ada").
  • A positive control in test 3: see "Inbox" and see "Queries: 0" run before the push, so the don't see steps can't pass on a text that never existed. N1 of the negative file sees "Inbox" before the push too, and fails only at the check after it (line 17).
  • NEXT.md item 2 says "back: in-app pop only (not the system back key or iOS edge swipe)".
  • Rerun after the change, on each device: navigation.probe passed 4 of 4 three times, and the negative file gave 0 passed and 5 failed, each with its stated message. Main is merged in; there was no conflict.

@pbertsch
pbertsch requested a review from a team as a code owner November 17, 2026 10:12
@pbertsch
pbertsch merged commit 74907c8 into main Nov 17, 2026
10 of 12 checks passed
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…d on two named devices (AlphaWaveSystems#792)

## Summary

**Widget family: Navigation (app bar and routes).** Item 2 of `docs/compat/NEXT.md`: a fixture screen, a ProbeScript suite, the variants that must fail, and the matrix results, set only from runs on named devices.

- **Screen** (`PROBE_FIXTURE_WIDGETS=navigation`, `native-test-apps/flutter/lib/widget_families.dart`):
  - a `Scaffold` whose `AppBar` is titled "Inbox", with an action `IconButton` with the tooltip "Search" (`wf_search`), counted in the state text `Queries: N`;
  - a button "Open message" (`wf_open_message`) that pushes a `MaterialPageRoute` titled "Message", whose body is `From: Ada`.

  No label is part of a state text, since text selectors match substrings; that is why the counter is `Queries` and not `Searches`. Every control has a `Semantics(identifier:)` and a `ValueKey` of the same name, as in the other families.
- **Suite** `probe-tests/widgets/navigation.probe` (4 tests): the bar's title, and its action by tooltip and by `#id`; a pushed route is seen; **the route under the top one is not seen**; `go back` returns to the first route.
- **Negative variants** `probe-tests/negative/navigation.probe.expect-fail` (5 tests, each with its `# Expected:` line): the route under the top one seen; one tap counted twice; the second route before it is pushed; the second route after `go back`; a tooltip that is not on the screen.
- **Matrix:** a Navigation section, and the family row now points to it. The cells are set only for `AppBar` and `Navigator`; `NavigationBar`, `TabBar` and `Drawer` stay not assessed. `NEXT.md` item 2 says it is assessed; the README lists the family. CHANGELOG under Added.

## What it settles

- **The route rule holds on a device.** While the message route (opaque) is on top, the first route's title, texts and button are not matched: `don't see "Inbox"`, `"Queries: 0"` and `"Open message"` pass, and `see "Inbox"` fails. This is the agent's rule (`_isVisible` and `_routeOnScreen` in `probe_agent/lib/src/finder.dart`), which until now had only a unit test (PT-03).
- **Finding: during a pop, the leaving route is still matched, on both devices.** Right after `go back`, `wait until "Inbox" appears` passes at once, while the message route is still on screen in Material's page transition. The first run of the suite failed at `don't see "From: Ada"`: it found 2 elements. Those steps alone, 5 runs on each device: the leaving route was found in 4 of 5 runs on `Compat-Pixel-API35` and in 5 of 5 on `Compat-iPhone`. A test that checks a pop now waits for the leaving route to go (`wait until "From: Ada" disappears`). The suite says this in a comment, and the negative N4 uses the same wait.
- `go back` is the agent calling `Navigator.pop()`. It is not the Android back key or the iOS edge swipe; neither was run.

## Evidence (devices)

Flutter **3.44.8** (installed; `pubspec.lock` unchanged, nothing added or upgraded). The probe CLI was built from this branch; the agent is the fixture's path dependency (0.27.0).

- **Android:** AVD `Compat-Pixel-API35` (created for this work from the installed `android-35/google_apis/arm64-v8a` image, Pixel 6 profile), Android 15 (API 35), `emulator-5590`. The name was read back (`adb -s emulator-5590 emu avd name`) before each step. Host port 48690, device port 48686.
- **iOS:** simulator `Compat-iPhone`, UDID `A57AEE67-3473-4C20-BE25-8847C2A9D6BE` (created for this work), iPhone 17 Pro, iOS 26.5, always addressed by UDID. The agent was built with `--dart-define=PROBE_PORT=48691`, its own port, since a simulator shares the host's network.

| Run | `Compat-Pixel-API35` | `Compat-iPhone` |
|---|---|---|
| `navigation.probe`, 3 runs | 4 passed, 3 times | 4 passed, 3 times |
| the negative file, run as its header says | 0 passed, 5 failed, each with its stated message | 0 passed, 5 failed, each with its stated message |

Flutter 3.50.0 was not run for this family; the matrix says so. I've asked where a 3.50.0 SDK copy is; if one is available, the suite runs on it in a follow-up and the cells name both versions.

## Not covered

`NavigationBar`, `TabBar`, `Drawer`; a route that is not opaque (a dialog, a popup menu, a sheet, where the page under it stays matched by design; item 8); named routes and `go_router`; the system back key and the iOS edge swipe; a nested `Navigator`; Cupertino routes (item 10); a physical device; other screen sizes, text scales and accessibility services. CI does not build the fixture.

## Test plan

- [x] The suite and the negative file on both named devices (table above).
- [x] `make ci-local`: 5 of 5 required checks at cc5bf40 (Build, Go unit tests, probe-convert tests, Lint, Disclosure guard).
- [x] `go test ./scripts/`: the negative-suite guards accept the new file.
- [x] `dart format` and `flutter analyze lib`: no issues.

## Review follow-up

- **The transition finding names its devices**, in the matrix, the CHANGELOG and here: Android 4 of 5 runs, iOS 5 of 5 (`pop-repro`: tap, wait for the message, `go back`, `wait until "Inbox" appears`, `don't see "From: Ada"`).
- **A positive control in test 3:** `see "Inbox"` and `see "Queries: 0"` run before the push, so the `don't see` steps can't pass on a text that never existed. N1 of the negative file sees `"Inbox"` before the push too, and fails only at the check after it (line 17).
- **`NEXT.md` item 2** says "back: in-app pop only (not the system back key or iOS edge swipe)".
- **Rerun after the change**, on each device: `navigation.probe` passed 4 of 4 three times, and the negative file gave 0 passed and 5 failed, each with its stated message. Main is merged in; there was no conflict.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant