Skip to content

docs(compat): a ranked plan of the next compatibility tests, from widget use in the Flutter sample apps - #759

Merged
pbertsch merged 2 commits into
mainfrom
docs/compat-next
Nov 17, 2026
Merged

pbertsch merged 2 commits into
mainfrom
docs/compat-next

Conversation

@pbertsch

@pbertsch pbertsch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR adds docs/compat/NEXT.md, a ranked plan of the compatibility tests to write next, and docs/compat/widget-usage.py, the count behind it. Documentation only: no matrix cell changes, and nothing in it claims how FlutterProbe behaves with a widget.

  • Ranking. For each widget in catalog.json (Flutter 3.50.0), the script counts how many of the 36 packages of flutter/samples construct it. The commit is pinned to 12a1916 (2026-11-11). Tests, generated files and comments are excluded. Layout-only widgets are excluded, apart from the ones that change what see or tap mean.
  • The 11 items. Text, then the app bar and routes, widgets without text, scrolling lists, tiles, progress indicators, gestures, dialogs, snack bars, the Cupertino bars and overlapping widgets. Each item names:
    • the fixture screen (PROBE_FIXTURE_WIDGETS=<family>);
    • the ProbeScript steps, all of which exist in the grammar today. The one follow-up that needs a missing verb (store text of a Flutter widget) says so;
    • the variants that must fail;
    • the open question the test settles.
  • Two open questions worth naming:
    • The route rule on a device. The agent does not match a route under an opaque one, and an agent unit test covers that (probe_agent/test/finder_test.dart, PT-03). Under a dialog, popup or sheet the page stays matched by design. What is open is a device run.
    • Does a lazy list's unbuilt row count as absent?
  • Also listed:
    • the follow-ups for the cells the matrix marks "partly";
    • the families the source cannot rank (pickers, menus, tabs, drawers, sheets are rare in demo apps), without a usage claim;
    • the limits of the method.
  • docs/compat/README.md lists the two files, and the CHANGELOG has the entry.

The newest stable Flutter in the official release manifest is still 3.50.0 (Dart 3.14.0, 2026-11-11), which RELEASES.md already records. Nothing to add there.

Test plan

  • The committed script, run on the same checkout, gives output byte-identical to the run the ranking was taken from (36 packages, 185 widgets used).
  • Every step form named in the plan exists in website/src/content/docs/probescript/grammar.md or in an existing suite.
  • The newest stable release was checked against the official manifest.

@pbertsch
pbertsch requested a review from a team as a code owner November 17, 2026 04:50
… dialog stays matched, and a Flutter field's text has no store verb
@pbertsch
pbertsch merged commit 0c11300 into main Nov 17, 2026
10 of 12 checks passed
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…get use in the Flutter sample apps (AlphaWaveSystems#759)

## Summary

This PR adds `docs/compat/NEXT.md`, a ranked plan of the compatibility tests to write next, and `docs/compat/widget-usage.py`, the count behind it. Documentation only: no matrix cell changes, and nothing in it claims how FlutterProbe behaves with a widget.

- **Ranking.** For each widget in `catalog.json` (Flutter 3.50.0), the script counts how many of the 36 packages of `flutter/samples` construct it. The commit is pinned to `12a1916` (2026-11-11). Tests, generated files and comments are excluded. Layout-only widgets are excluded, apart from the ones that change what `see` or `tap` mean.
- **The 11 items.** `Text`, then the app bar and routes, widgets without text, scrolling lists, tiles, progress indicators, gestures, dialogs, snack bars, the Cupertino bars and overlapping widgets. Each item names:
  - the fixture screen (`PROBE_FIXTURE_WIDGETS=<family>`);
  - the ProbeScript steps, all of which exist in the grammar today. The one follow-up that needs a missing verb (`store text of` a Flutter widget) says so;
  - the variants that must fail;
  - the open question the test settles.
- **Two open questions worth naming:**
  - The route rule on a device. The agent does not match a route under an opaque one, and an agent unit test covers that (`probe_agent/test/finder_test.dart`, PT-03). Under a dialog, popup or sheet the page stays matched by design. What is open is a device run.
  - Does a lazy list's unbuilt row count as absent?
- **Also listed:**
  - the follow-ups for the cells the matrix marks "partly";
  - the families the source cannot rank (pickers, menus, tabs, drawers, sheets are rare in demo apps), without a usage claim;
  - the limits of the method.
- `docs/compat/README.md` lists the two files, and the CHANGELOG has the entry.

The newest stable Flutter in the official release manifest is still 3.50.0 (Dart 3.14.0, 2026-11-11), which `RELEASES.md` already records. Nothing to add there.

## Test plan

- [x] The committed script, run on the same checkout, gives output byte-identical to the run the ranking was taken from (36 packages, 185 widgets used).
- [x] Every step form named in the plan exists in `website/src/content/docs/probescript/grammar.md` or in an existing suite.
- [x] The newest stable release was checked against the official manifest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant