Skip to content

chore(vscode): regenerate keywords and snippets; ci-local runs the extension tests for ProbeScript doc changes - #815

Merged
pbertsch merged 1 commit into
mainfrom
chore/vscode-regenerate-keywords
Nov 17, 2026
Merged

pbertsch merged 1 commit into
mainfrom
chore/vscode-regenerate-keywords

Conversation

@pbertsch

Copy link
Copy Markdown
Member

Summary

Main is red on two VS Code extension checks: "the generated keyword files are up to date with the docs" and "the generated snippet files are up to date with the docs". #806 changed the see exactly entry in website/src/content/docs/probescript/dictionary.md and didn't regenerate the extension's files from it.

  • Regenerated with npm run gen:keywords and npm run gen:snippets in vscode/. The diff is one line in each file: the see exactly snippet description, and the docs hash in probescriptKeywords.generated.ts.
  • Why it wasn't caught: scripts/ci-local.sh adds the Studio frontend check to a default run when the change touches the ProbeScript docs. It didn't add the VS Code extension tests, which are made from the same docs, and the hosted workflow only runs for changes under studio/ and vscode/. Both checks are now added for those paths, and the run's note says so. --all and --job are unchanged.

No CHANGELOG entry: generated files and local CI tooling only, with nothing user-facing.

Test plan

  • npm test in vscode/: 84 of 84 pass (the two checks fail on main).
  • go test ./scripts/ passes. ci_local_plan_test.go now expects both checks, and the note, for each docs input. It expects neither check for other paths, and each check once under --all with no note.
  • Mutant: adding only the frontend, as before, fails the plan test.

…tension tests when the ProbeScript docs change
@pbertsch
pbertsch requested a review from a team as a code owner November 17, 2026 16:04
@pbertsch
pbertsch merged commit 8654dfb into main Nov 17, 2026
10 of 11 checks passed
jay23606 added a commit to jay23606/flutter-probe that referenced this pull request Nov 17, 2026
…tension tests for ProbeScript doc changes (AlphaWaveSystems#815)

## Summary

Main is red on two VS Code extension checks: "the generated keyword files are up to date with the docs" and "the generated snippet files are up to date with the docs". AlphaWaveSystems#806 changed the `see exactly` entry in `website/src/content/docs/probescript/dictionary.md` and didn't regenerate the extension's files from it.

- **Regenerated** with `npm run gen:keywords` and `npm run gen:snippets` in `vscode/`. The diff is one line in each file: the `see exactly` snippet description, and the docs hash in `probescriptKeywords.generated.ts`.
- **Why it wasn't caught:** `scripts/ci-local.sh` adds the Studio frontend check to a default run when the change touches the ProbeScript docs. It didn't add the VS Code extension tests, which are made from the same docs, and the hosted workflow only runs for changes under `studio/` and `vscode/`. Both checks are now added for those paths, and the run's note says so. `--all` and `--job` are unchanged.

No CHANGELOG entry: generated files and local CI tooling only, with nothing user-facing.

## Test plan

- [x] `npm test` in `vscode/`: 84 of 84 pass (the two checks fail on main).
- [x] `go test ./scripts/` passes. `ci_local_plan_test.go` now expects both checks, and the note, for each docs input. It expects neither check for other paths, and each check once under `--all` with no note.
- [x] Mutant: adding only the frontend, as before, fails the plan test.
pbertsch added a commit that referenced this pull request Nov 17, 2026
…e in the terminal (#813)

## Summary

The VS Code extension typed each tool's command line into the integrated terminal, quoted for the terminal's shell. **That can't be made safe**, so this PR removes it: the extension now runs the binary itself, with its arguments as a list and no shell, and shows the output in its own terminal.

**Why quoting can't work.** The review reproduced this, and I reproduced it again. Under bash with `LC_ALL=C` or `LC_ALL=POSIX` (an SSH session, a container, any terminal with no locale), readline reads the bytes of a UTF-8 character one by one and acts on some of them as keys. The second byte of `Ò` (U+00D2, `C3 92`) and of U+2012 (`E2 80 92`) is `0x92`, which readline treats as revert-line: it drops everything typed so far, the opening quote included, and the rest is read as a new line.

I typed the old code's quoted argument for `aÒx;touch PWN8;'` into an interactive shell in a real pty (`printf '%s\n' <word>`, then Enter):

| Shell, locale | `touch PWN8` ran |
|---|---|
| `/bin/bash` (3.2.57), `LC_ALL=C` | **yes** |
| `/bin/bash`, `LC_ALL=POSIX` | **yes** |
| `/bin/bash`, `en_US.UTF-8` | no |
| `/bin/zsh`, `LC_ALL=C` | no |

The locale of the terminal can't be detected from the extension, and the same holds for other bytes and line editors. The earlier version of this PR, and `shellEscape` before it, had the same hole.

## The change

- **`vscode/src/terminal/processRun.ts` (new).**
  - `RunQueue` starts the binary with `spawn(command, args, { shell: false, … })`. No shell parses, expands or line-edits an argument, on any system, under any terminal shell or locale.
  - It writes the output with `\r\n` line ends.
  - A run asked for while another runs in the same terminal starts when that one ends, as a command typed into a busy shell did.
  - Typed text is echoed and sent to the running tool line by line. Backspace edits the line, and arrow and function keys are dropped rather than typed as their letters. Ctrl+C sends SIGINT.
  - Closing the terminal drops waiting runs and sends SIGINT. Stopping a recording disposes its terminal, so `probe record` still gets its interrupt.
  - The command is shown above its output. Control characters, line and paragraph separators and bidirectional characters are written as `\uXXXX`, so a file name can't move the cursor, recolor the terminal or reorder what's shown.
- **`probeTerminal.ts`.** Each named terminal is a VS Code `Pseudoterminal` backed by a `RunQueue`. Output written before VS Code opens the terminal is kept until `open`, because the API drops earlier events. The output stays after the run ends.
- **Removed.** `argWord`, `commandWord`, `quoteArg`, `quotePosix` and `refuseControl` are gone, along with the shell-name logic. So are `terminalAction` and `envFingerprint`, which only existed because a terminal's environment was fixed when it was created; each run now gets its own environment. The secret-flag refusal stays.
- **Why not `createTerminal({ shellPath, shellArgs })`.** That would also skip the shell, but each run would get a new terminal process, and keeping the output after the run ends depends on the user's settings. The pseudoterminal keeps the output, reuses the terminal, and queues runs.

**What users notice:**

- The run gets VS Code's environment (plus the secrets of a test run, as before), not the one their shell profile builds. A variable set only in a profile that VS Code didn't read won't reach it.
- The command isn't in the shell's history.
- The terminal isn't a shell to type other commands into.

This is in the CHANGELOG and the README.

**The other review points** applied to the quoting code, which is removed:

- zsh's `=ls`: `=ls` is now just an argument, and it's in the hostile set;
- the cmd-branch mutant;
- PowerShell 5.1 dropping an empty argument: an empty argument now reaches the tool, and it's in the hostile set;
- `!` under cmd's delayed expansion.

The CHANGELOG line about control characters is replaced by the new entry. The #773 entry now points to it.

## Tests (`vscode/src/test/processRun.test.ts`, 10)

- **Real processes.** Node runs a script that prints its arguments. 21 hostile arguments reach it as 21 arguments, unchanged, and no `PWNED` file appears. They include the quote and `$(…)` payloads, the backtick, `%PATH%`, `=ls`, an empty argument, a trailing backslash, a newline, bidi characters, and the `Ò` and U+2012 payloads.
- **The review's case.** The `Ò` payload under `LC_ALL=C` and under `LC_ALL=POSIX` arrives as one unchanged argument, and `PWN8` isn't created.
- **Errors.** A binary that can't be started says so (ENOENT), and the next run still starts.
- **Queue behavior, with a stand-in child:**
  - a second run waits, then starts;
  - the run's cwd and env are passed;
  - output gets `\r\n`, and an error followed by a close ends the run once;
  - echo, a line sent to the tool, backspace, arrow and F1 dropped, Ctrl+C → SIGINT;
  - closing the terminal drops waiting runs and interrupts;
  - the shown command escapes ESC, bidi characters, C1 characters, quotes and backslashes.
- **Source checks** (the terminal file imports `vscode`):
  - output is buffered until `open`;
  - `close` stops the queue and typed text goes to the queue;
  - no `sendText`, `shellPath` or `child_process` in the terminal file;
  - `spawn` uses `shell: false`.

## Mutation check

Each mutant was applied, compiled and tested, then restored byte for byte. All 13 were killed:

- spawning through a shell;
- escape sequences typed as letters;
- Ctrl+C not interrupting;
- the interrupt being a plain kill;
- a second run starting at once;
- `stop` keeping waiting runs;
- a run ending twice;
- bare `\n` written to the terminal;
- control characters shown raw;
- the run's env not added;
- output before `open` dropped;
- closing not stopping the run;
- the secret flag not refused.

## Test plan

- [x] The extension's tests: 82 of 82 pass. This includes the generated-file checks, now green after #815. A local `npm test` here also picks up a stale compiled `out/test/argWord.test.js` from before its source was removed; `out/` is ignored, and a fresh checkout doesn't have it.
- [x] `tsc` is clean, and no raw control or format characters are in the sources.
- [x] The old hole was reproduced with a real pty, as above.
- [ ] Manual run in VS Code (Run test, Record and stop, Devices): not done here, since no VS Code UI session is available to me.

## Second review follow-up

- **Blocker: orphaned processes.** `stop()` (closing the terminal, or deactivating the extension) only sent SIGINT.
  - **macOS and Linux:** the tool now starts in its own process group (`detached`), and SIGINT goes to the whole group, as Ctrl+C in a shell does. A tool still running after 15 seconds gets SIGKILL (`probe record` waits up to 5 s for the agent after SIGINT, then writes the file), with its group. The timer is cancelled when the tool ends.
  - **Windows:** there is no Ctrl+C for another process, so Ctrl+C and close both end the tool and every program it started with `taskkill /T /F /PID`, run by its full path under `%SystemRoot%\System32`. The README now says that a recording isn't saved there.
- **EPIPE:** `child.stdin` has an error handler, so typing into a tool that closed its input isn't an uncaught error.
- **README:** the secrets paragraph now describes per-run environments; the old text said they stayed in the terminal for every command typed there. It also covers what Ctrl+C does on each OS, and notes that the tool's output, escape sequences included, is written as it is.
- **The two test gaps from the review:**
  - the run's env now wins over VS Code's own `PROBE_CLOUD_KEY`, with a test;
  - a spawn that throws still starts the next waiting run, tested both as the first run and while another run waits.
- **`.bat` and `.cmd` on Windows** are refused up front with a clear message, which modern Node would otherwise turn into an EINVAL. **Not verified on a real Windows machine:** I have none here. The Windows branches are covered by unit tests with the platform injected.
- **Caps:** 100 waiting runs, a typed line of 64 KiB, and 1 MiB of output kept before the terminal opens. Beyond that, the output is cut with a note.
- Main is merged. The CHANGELOG conflict is resolved, and this PR's entry is last in Security.

**Tests: 8 new**, 18 in `processRun.test.ts`.

- **Real processes (POSIX).** These are the review's `o.js` case:
  - a tool that ignores SIGINT and starts `sleep 300`: after `stop()` with a 300 ms grace period, both are gone 1.2 s later, and the waiting run doesn't start;
  - Ctrl+C ends the `sleep` grandchild through the process group;
  - a tool that closes its input while Enter is typed twice: no uncaught exception (the review's `e.js` case).
- **With the stand-in child:**
  - env precedence;
  - a spawn that throws;
  - SIGINT, then SIGKILL after the grace period, and no SIGKILL when the tool ended in time;
  - the `.bat` and `.cmd` refusal on Windows only;
  - the caps.

**Mutants:** 10 of 11 killed. The 11th, not cancelling the SIGKILL timer when the tool ends, is equivalent: the timer only signals while `this.running === child`, so removing the `clearTimeout` changes nothing observable. The extension tests pass, 90 of 90.

## Third review follow-up

- **Deactivate could orphan a tool.** `deactivate()` returned at once, and the 15-second SIGKILL timer died with the extension host.
  - `RunQueue.shutdown(graceMs)` sends SIGINT, sends SIGKILL after `graceMs`, and resolves once the tool has ended.
  - `deactivate()` now returns `stopAllTerminals()`, which shuts every queue down with a 1-second grace and waits for them, capped at 3 seconds. That's a short time because the window is closing and VS Code waits only briefly; I didn't verify how long VS Code waits.
  - A recording in progress isn't saved on deactivate. The README and CHANGELOG say so.
  - A later `stop()` (the terminal's own close) never moves a SIGKILL later than one already scheduled.
- **README and CHANGELOG:** if the extension host crashes, a tool it started keeps running until it next writes output, when its closed pipe ends it.
- **Tests for the survivors:**
  - `stop()` of a run that already ended signals nothing;
  - the 15-second default grace, pinned with mock timers (14,999 ms: SIGINT only; 15,000 ms: SIGKILL);
  - the `taskkill` arguments `/T /F /PID <pid>` by full path under `SystemRoot`, with a fallback to Node's kill when `taskkill` can't start, through the new `makeSignalTree` with a stand-in spawn;
  - a refused `.cmd` lets the next waiting run start;
  - after a stopped run ends, the next run still escalates;
  - deactivate's wiring (source check);
  - the review's `d.js` case as a real process: a tool that ignores SIGINT is gone when `shutdown(300)` resolves.
- **Mutants: 11 of 12 killed.** The survivor is equivalent. Removing the timer's `this.running === child` guard can't change a result while the `clearTimeout` on exit exists; the two are redundant defences against signalling a reused pid. My first run of this set hung on the mutant whose `shutdown` never resolves, so the harness now runs with `--test-timeout`. I ended that hung test process by its pid.
- **Results:** `processRun.test.ts` passes 26 of 26, and the extension tests pass 98 of 98.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant