Test specification and harness - #27
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a standardized “tests-as-data” harness for JSON Grammars, enabling grammar test suites to be expressed as portable JSON and executed via a pure runner, a CLI, or a Jest adapter.
Changes:
- Added a portable JSON test-suite specification (TS types + JSON Schema) and example corpora for the JSON and JSON-expression grammars.
- Implemented a grammar driver/runner/reporting stack plus adapters for CLI execution and Jest integration.
- Documented the testing approach and added a
grammar-testscript for snapshot regeneration.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/testing/types.ts | Defines the portable test suite/case data model and channel semantics. |
| src/testing/runner.ts | Core suite/case execution, snapshot update behavior, and result shaping. |
| src/testing/report.ts | Human-readable formatting for runner results. |
| src/testing/jest.ts | Jest adapter to run JSON suites as native Jest tests. |
| src/testing/index.ts | Barrel exports for the testing harness modules. |
| src/testing/grammar-test.schema.json | JSON Schema for validating suite JSON files. |
| src/testing/driver.ts | TypeScript “driver” surface: compilation, execution, CST projection, input generation. |
| src/testing/cli.ts | Standalone CLI runner with --update snapshot writing. |
| src/testing/tests/harness.spec.ts | Unit tests for harness behavior (channels, snapshots, only/skip, generation). |
| src/grammars/tests/json.test.json | JSON grammar test corpus in the new standardized format. |
| src/grammars/tests/json-expression.test.json | JSON-expression grammar test corpus in the new standardized format. |
| src/grammars/tests/harness.spec.ts | Wires the JSON corpora into Jest via jestSuiteFile. |
| README.md | Adds a “Testing” section linking to the new docs. |
| package.json | Adds a grammar-test script for running/updating suites. |
| docs/testing.md | New documentation explaining the portable test format and workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+16
to
+22
| const show = (value: unknown): string => { | ||
| if (Array.isArray(value) && value.every((v) => typeof v === 'string')) { | ||
| // Printed-tree channels (cstPrint/trace): show as the rendered tree. | ||
| return '\n' + (value as string[]).map((l) => ' ' + l).join('\n'); | ||
| } | ||
| return JSON.stringify(value); | ||
| }; |
Comment on lines
+54
to
+56
| case 'ast': | ||
| return d.ast; | ||
| case 'cst': |
Comment on lines
+109
to
+114
| * How to generate an input for a case. `"sample"` is shorthand for | ||
| * `{sample: true}` — drive generation from each rule's `sample` field, which is | ||
| * deterministic wherever samples are defined. A `seed` makes random generation | ||
| * reproducible; `count` repeats the case with fresh inputs. | ||
| */ | ||
| export type GenerateSpec = 'sample' | {sample?: boolean; seed?: number; count?: number}; |
Comment on lines
+69
to
+78
| export const resolveInput = (grammar: Grammar, tc: TestCase): string => { | ||
| if (tc.src !== undefined) return tc.src; | ||
| if (tc.generate !== undefined) { | ||
| const spec = tc.generate === 'sample' ? {sample: true} : tc.generate; | ||
| const rnd = spec.seed !== undefined ? mulberry32(spec.seed) : undefined; | ||
| const generator = new Generator({grammar, useSamples: spec.sample !== false, rnd}); | ||
| return generator.gen(tc.rule ?? grammar.start); | ||
| } | ||
| return ''; | ||
| }; |
| same test files work across programming languages and across test runners, so | ||
| you develop a grammar by editing just two things: the grammar and its tests. | ||
|
|
||
| See [docs/testing.md](docs/testing.md) for a introduction. |
| /** Full suite rendering with a trailing summary line. */ | ||
| export const formatSuite = (res: SuiteResult, useColor = true): string => { | ||
| const lines: string[] = []; | ||
| if (res.describe) lines.push(color(useColor, '', res.describe)); |
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.