Skip to content

feat(fixtures): Lists and scrolling widget family: screen, suite, negative variants and matrix results - #798

Merged
pbertsch merged 4 commits into
mainfrom
feat/fixture-family-lists
Nov 17, 2026
Merged

pbertsch merged 4 commits into
mainfrom
feat/fixture-family-lists

Conversation

@pbertsch

@pbertsch pbertsch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

The Lists and scrolling widget family: a fixture screen, a suite, negative variants, and the matrix cells and findings, in one pull request (the second of three families; the next is Dialogs and sheets).

  • Screen (PROBE_FIXTURE_WIDGETS=lists, native-test-apps/flutter/lib/widget_families.dart): a lazy ListView.builder of 200 rows, with a page each for a GridView.count (60 cells), a CustomScrollView with a SliverList (200 items), a SingleChildScrollView (40 lines and a button), and tiles (Card holding a ListTile with a title, a subtitle and a trailing icon; a CupertinoListTile). Every element has a Semantics identifier and a ValueKey, and every state is also shown as text.
  • Suite probe-tests/widgets/lists.probe, 11 tests, and negative variants probe-tests/negative/lists.probe.expect-fail, 9 tests that must fail, each with its expected message in a # Expected: comment (covered by the guard of feat(fixtures): Buttons widget family: screen, suite, negative variants and matrix results #746).
  • Matrix (docs/compat/MATRIX.md): the family row, a section with a table and findings, the how-it-was-run paragraph; docs/compat/NEXT.md items 4 and 5; the fixture README; CHANGELOG.

Findings (all measured, in the matrix)

  • A lazy list, grid or sliver list builds the rows near the top only: see and tap do not find the others until scroll down until builds them. scroll down until a target that does not exist ends with "not found after scrolling" and does not run on.
  • A SingleChildScrollView builds every child: tap "Bottom button" scrolls to it by itself.
  • A text selector takes the first widget that holds the text, so a name that a state text also holds selects the state text: after tap "Alice", long press "Alice" lands on Tapped: Alice, not on the tile. (My first draft of the suite failed on exactly this; it is now a test.)
  • A tile's title is not a container of its tile: see "Online" in "Alice" fails; "Online" below "Alice" works.
  • A line of a SingleChildScrollView that is built but below the fold is not seen (see "Line 39" fails), while tap "Bottom button" still scrolls to a built button.
  • see exactly 1 "<one Text>" reports "found 2" on both Flutter versions (a row title, a state text and a button label, each measured): the finder matches both the Text and the RichText it builds. It is pinned by negative variant N9 (which must be flipped or removed when the agent is fixed) and not fixed here.

Checked

  • Device: emulator-5600, AVD ProbeWebViewSpike (Android 15, API 35; the name was read back before each install and run). A build by Flutter 3.44.8 and one by 3.50.0 (the 3.50.0 build in a scratch copy with a private SDK): lists.probe 10 of 10 on each; the negative file 0 passed, 9 failed on each, every test at its own step with the message in its comment. After review edits (a test of a line below the fold, a precondition don't see "Bottom button", quoted messages) and a merge of main: 11 of 11 and 0 of 9 again, on the 3.44.8 build on the same emulator (the 3.50.0 build was not re-run).
  • Before the final suite, small diagnostic files measured each behavior above; the failing ones became the findings and negatives.
  • flutter analyze on the fixture screen: no issues. The new code was formatted in a temporary file (the existing file was not reformatted).
  • make ci-local: 5 of 5 required checks. Disclosure scan clean. A fresh clone of the branch has the new files, the guard test passes, and its fixture source equals the one that was built and run.

Not checked / limits

  • One device and one Android version; no iOS, no physical device, no TalkBack, no other screen sizes or text scales.
  • Not covered by the screen: PageView, Scrollbar, a horizontal list, two scrollables on one screen, ListView.separated, pull to refresh, a list inside a dialog, a list under an open keyboard, scroll by an amount.
  • In some earlier small diagnostic runs the first test failed in before each test ("restart the app: reconnect with pre-shared token: agent not reachable within 20s"); it did not happen in the full-suite runs reported here and is under investigation (a follow-up of issue Steps right after restart the app can run before the first frame has settled on slow devices #793).
  • No mutants of the fixture: the 9 negative variants are the check that the positive tests can fail.

Test plan

  • lists.probe 10 of 10 on a Flutter 3.44.8 build and a 3.50.0 build (emulator-5600); 11 of 11 on 3.44.8 after the review edits
  • negative file 0 of 9 passing on both builds, each at its own step
  • make ci-local 5 of 5, scan clean, fresh clone checked
  • Review of every matrix cell against the suite

🤖 Generated with Claude Code

https://claude.ai/code/session_01YXQ37Hv3SpnR5WWWmGy2VD

…ative variants and matrix results

A lists screen (PROBE_FIXTURE_WIDGETS=lists): a lazy ListView.builder of 200 rows with a page each for a GridView.count, a CustomScrollView with a SliverList, a SingleChildScrollView and tiles (Card with ListTile, CupertinoListTile). lists.probe (10 tests) and its 9 negative variants ran on builds by Flutter 3.44.8 and 3.50.0 on one Android emulator: 10 of 10, and 0 of 9. The matrix has the cells and the findings: an unbuilt row is neither seen nor tapped until a scroll builds it, a scroll to a missing target ends with a failure, a text selector takes the first widget that holds the text, a title is not a container of its tile, and see exactly counts a Text twice.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YXQ37Hv3SpnR5WWWmGy2VD
pbertsch and others added 3 commits November 17, 2026 08:06
…p note, a below-the-fold test, a precondition

N3 quotes the real message of scroll down until, and N9 says it must be flipped or removed once the agent counts a Text once. A new test pins that a built line below the fold of a SingleChildScrollView is not seen, and the tap test of its bottom button has the precondition don't see "Bottom button". The restart note says the failure is under investigation.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YXQ37Hv3SpnR5WWWmGy2VD
…e lists negatives

The guard of the negative suites wants the "# Expected:" line directly before each test: the flip note of N9 goes above it. Comments only; the tests are unchanged.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YXQ37Hv3SpnR5WWWmGy2VD
@pbertsch
pbertsch merged commit 54d248d into main Nov 17, 2026
10 of 12 checks passed
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…ative variants and matrix results (AlphaWaveSystems#798)

## Summary

The Lists and scrolling widget family: a fixture screen, a suite, negative variants, and the matrix cells and findings, in one pull request (the second of three families; the next is Dialogs and sheets).

- **Screen** (`PROBE_FIXTURE_WIDGETS=lists`, `native-test-apps/flutter/lib/widget_families.dart`): a lazy `ListView.builder` of 200 rows, with a page each for a `GridView.count` (60 cells), a `CustomScrollView` with a `SliverList` (200 items), a `SingleChildScrollView` (40 lines and a button), and tiles (`Card` holding a `ListTile` with a title, a subtitle and a trailing icon; a `CupertinoListTile`). Every element has a `Semantics` identifier and a `ValueKey`, and every state is also shown as text.
- **Suite** `probe-tests/widgets/lists.probe`, 11 tests, and **negative variants** `probe-tests/negative/lists.probe.expect-fail`, 9 tests that must fail, each with its expected message in a `# Expected:` comment (covered by the guard of AlphaWaveSystems#746).
- **Matrix** (`docs/compat/MATRIX.md`): the family row, a section with a table and findings, the how-it-was-run paragraph; `docs/compat/NEXT.md` items 4 and 5; the fixture README; CHANGELOG.

## Findings (all measured, in the matrix)

- A lazy list, grid or sliver list builds the rows near the top only: `see` and `tap` do not find the others until `scroll down until` builds them. `scroll down until` a target that does not exist ends with "not found after scrolling" and does not run on.
- A `SingleChildScrollView` builds every child: `tap "Bottom button"` scrolls to it by itself.
- A text selector takes the first widget that holds the text, so a name that a state text also holds selects the state text: after `tap "Alice"`, `long press "Alice"` lands on `Tapped: Alice`, not on the tile. (My first draft of the suite failed on exactly this; it is now a test.)
- A tile's title is not a container of its tile: `see "Online" in "Alice"` fails; `"Online" below "Alice"` works.
- **A line of a `SingleChildScrollView` that is built but below the fold is not seen** (`see "Line 39"` fails), while `tap "Bottom button"` still scrolls to a built button.
- **`see exactly 1 "<one Text>"` reports "found 2"** on both Flutter versions (a row title, a state text and a button label, each measured): the finder matches both the `Text` and the `RichText` it builds. It is pinned by negative variant N9 (which must be flipped or removed when the agent is fixed) and not fixed here.

## Checked

- Device: emulator-5600, AVD `ProbeWebViewSpike` (Android 15, API 35; the name was read back before each install and run). A build by Flutter 3.44.8 and one by 3.50.0 (the 3.50.0 build in a scratch copy with a private SDK): `lists.probe` **10 of 10** on each; the negative file **0 passed, 9 failed** on each, every test at its own step with the message in its comment. After review edits (a test of a line below the fold, a precondition `don't see "Bottom button"`, quoted messages) and a merge of main: **11 of 11** and **0 of 9** again, on the 3.44.8 build on the same emulator (the 3.50.0 build was not re-run).
- Before the final suite, small diagnostic files measured each behavior above; the failing ones became the findings and negatives.
- `flutter analyze` on the fixture screen: no issues. The new code was formatted in a temporary file (the existing file was not reformatted).
- `make ci-local`: 5 of 5 required checks. Disclosure scan clean. A fresh clone of the branch has the new files, the guard test passes, and its fixture source equals the one that was built and run.

## Not checked / limits

- One device and one Android version; no iOS, no physical device, no TalkBack, no other screen sizes or text scales.
- Not covered by the screen: `PageView`, `Scrollbar`, a horizontal list, two scrollables on one screen, `ListView.separated`, pull to refresh, a list inside a dialog, a list under an open keyboard, scroll by an amount. 
- In some earlier small diagnostic runs the first test failed in `before each test` ("restart the app: reconnect with pre-shared token: agent not reachable within 20s"); it did not happen in the full-suite runs reported here and is under investigation (a follow-up of issue AlphaWaveSystems#793).
- No mutants of the fixture: the 9 negative variants are the check that the positive tests can fail.

## Test plan

- [x] `lists.probe` 10 of 10 on a Flutter 3.44.8 build and a 3.50.0 build (emulator-5600); 11 of 11 on 3.44.8 after the review edits
- [x] negative file 0 of 9 passing on both builds, each at its own step
- [x] `make ci-local` 5 of 5, scan clean, fresh clone checked
- [ ] Review of every matrix cell against the suite
pbertsch added a commit that referenced this pull request Nov 17, 2026
…erts the wrong count (#821)

## Summary

Follow-up of #798 and #806. The agent of main now counts one visible `Text` once (issue #800, #806), so negative variant N9 of the Lists suite, which pinned the old double count, would pass. This flips it as agreed:

- `lists.probe` has a new test, `see exactly counts a visible Text once`: `see exactly 1` of a row title, a state text and a button label (12 tests).
- N9 of `lists.probe.expect-fail` now asserts the wrong count: `see exactly 2 "Row 0"` must fail with "found 1".
- `docs/compat/MATRIX.md` and CHANGELOG say that the finding is fixed and what the suite does now.

## Checked

- Device: emulator-5600, AVD `ProbeWebViewSpike` (name read back before each run); the fixture is built with the agent of this branch (a path dependency), so it has the fix.
  - **Flutter 3.50.0**: `lists.probe` **12 of 12**; negative file **0 passed, 9 failed**, N9 at its own step with `Expected exactly 2 "text("Row 0")" but found 1`.
  - **Flutter 3.44.8**: `lists.probe` in 3 runs, 11 of 12 in each, the twelfth test not running because `before each test` failed to restart the app ("reconnect with pre-shared token: agent not reachable within 20s"), a different test each time; every test passed in at least one run, but not all 12 in one run. Negative file in 3 runs: 0 passed, 9 failed in each; in 2 of the 3 runs one test failed in `before each test` (the same restart failure) instead of at its own step, and in the third run every test failed at its own step with N9 ending in "found 1".
- `make ci-local`: 5 of 5; disclosure scan clean.

## Not checked / limits

- The restart failure is tracked in issue #823 (measurements per Flutter build, the `probe.yaml` of the runs; the machine load was not measured): in the runs above it hit every full run on 3.44.8, and 12 of 12 was reached on 3.50.0 in the runs made at the time; later runs on 3.50.0 (made for another pull request) had the same failure, so it is not specific to one Flutter version.
- One device, one Android version; no iOS.
pbertsch added a commit that referenced this pull request Nov 18, 2026
…ate bullet in Unreleased (#872)

## Summary
- Removes the duplicated bullet "The previews before an AI request can be used with the keyboard" from the Unreleased part of `CHANGELOG.md` (the #843 merge kept an older and a newer wording; the older, shorter one is removed, the longer one has the review's additions).
- Adds `scripts/changelog_duplicates_test.go`: a test that fails when two bullets of one section of the Unreleased part share the same bold title. A title check rather than "identical lines", because the copies are rarely byte for byte the same (the previews pair was not).
- Two older pairs were already there: the fixture app's "lists screen" bullet (#798 and its follow-up #821) and `store text ... in webview` (two revisions). They are not mine and which wording stays is their owners' call, so they are listed in `knownDuplicates` with the reason; the test fails if a known pair is resolved and its entry is left, and for any new duplicate.

## Test plan
- [x] A scan of the whole file found no other same-title bullet in the Unreleased part, and no byte-identical bullet anywhere
- [x] Tests: the detection (same title in one section; the same title in another section and in a released version are not duplicates; distinct bullets), the allowlist logic (an unknown duplicate fails, a resolved known one fails, all known passes); mutants killed: all versions checked, sections ignored, a pair no longer counted, known list ignored, gone entries ignored
- [x] guard 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant