Request: mark test runner stable in Node 20.0.0

[cjihrig]

What is the problem this feature will solve?

Having a stable test runner in core. People have been asking for it, and I feel the majority of the test runner is ready.

What is the feature you are proposing to solve the problem?

I would like to stabilize the test runner for the 20.0.0 release. This currently would not include code coverage, which is behind a separate flag.

What alternatives have you considered?

Not stabilizing the test runner.

[MoLow]

+1

[bnoordhuis]

the majority of the test runner is ready

I haven't been tracking closely but I get the impression people open feature requests for it quite a bit. How would moving it out of experimental interact with those?

[cjihrig]

Feature requests would have to follow semver. Of course we aren't obligated to implement all of the feature requests too - there are a few open feature requests that I personally don't think we should implement.

[GeoffreyBooth]

cc @nodejs/test_runner

I think this should be fine. We could just create experimental flags or warnings for any parts that are remaining experimental or that we think might get breaking changes. So really that’s my only question: are there any parts of the test runner, besides code coverage, that might get breaking changes? Reporters? How about we isolate those under experimental flags so that it’s clear to users when they’re deviating from the stable path, and call the rest stable.

[ljharb]

Is there a way we could measure "ready" beyond just "i feel"?

It'd be great to try to collect feedback about who has migrated to node's test runner, and who hasn't and why not.

[mcollina]

I think the test api and runner are stable. The reporters and codecov are too new to be stable, however they have a very little (if at all) API surface.

I’d recommend we keep the codecov and reporters as doc-experimental until 20.0.0 reach LTS but otherwise drop all flags and warnings.

From a marketing perspective, announcing v20 with test runner is a great announcement.

[meyfa]

I'd like to bring up #44023 again. I haven't cared about this topic much since opening that issue due to being busy with other things, but I consider that problem largely unsolved. The PR that closed that issue (#44241) is nice for sure but doesn't address command-line usage, and so IMHO doesn't really improve the TypeScript DX that much. Stabilizing could prevent such improvements from happening since they'd possibly be breaking changes. Please reconsider adding support for a string argument to --test, or some other way of non-programmatically specifying the test file pattern, before marking as stable.

To recap: The test runner has hardcoded globs for the test files. It doesn't support running test files written in TypeScript or any other language, or otherwise filtering the test files, without (1) listing all of them explicitly, or (2) writing a custom entry point. (1) is near impossible to maintain, and (2) requires significant amounts of boilerplate code and/or third-party packages, which defeats the point of a native test runner.

[mcollina]

@meyfa could you open a new issue with the things that are missing? Note that marking it stable doesn't mean we cannot add things, but rather that we won't break users.

[MoLow]

there is still an open issue on the subject #46292.
Although changing the behavior of --test can be a breaking change, I think that adding glob support into core (which I am very much in favor of) will probably take time and should not delay the test runner itself

[mcollina]

Marking tsc-agenda for visibility.

[GeoffreyBooth]

Although changing the behavior of --test can be a breaking change, I think that adding glob support into core (which I am very much in favor of) will probably take time and should not delay the test runner itself

Would adding glob support be a breaking change in some way? Other than “before it errored and now it doesn’t.”

As far as I’m concerned we should identify any parts that might still get breaking changes and wall those off as experimental, and mark the remainder as stable. I also think we should add warnings or experimental flags for the parts that might still change, because with the bulk of the test runner being stable it shouldn’t fall on users to pore through the docs to ensure that they’re only using the stable parts of the runner. I would imagine that there would likely be many users who only want to use the stable parts, waiting for the experimental parts to become stable, because they don’t want to deal with updating their testing code often as the experimental parts change.

[aduh95]

Would adding glob support be a breaking change in some way? Other than “before it errored and now it doesn’t.”

It would be a breaking change if we use * to mean "all the files" when before it meant a specific file. E.g.:

$ rm -rf repro && mkdir repro && cd repro
$ mkdir "**"
$ echo 'console.log("pass")' > "**/*.js"
$ echo 'throw new Error' > "**/should-fail.js"
$ node --test **/*.js
TAP version 13
# Subtest: …/repro/**/*.js
    # pass
ok 1 - …/repro/**/*.js
  ---
  duration_ms: 112.994764
  ...
1..1
# tests 1
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 115.977816
$ cd .. && rm -r repro

If we change the meaning of **/*.js to say "all the .js files in all the subdirectories", it would now execute **/should-fail.js and the test suite would start failing.