ci: upload junit formatted test results by cpuguy83 · Pull Request #5271 · containerd/containerd

cpuguy83 · 2021-03-25T19:12:56Z

ci: upload junit formatted test results

This allows us to dig more details out of test runs and maintain a
better history.

For this we can use gotestsum, which is a utility that wraps go test
so that it outputs test2json (go's format) and output junit (a format
more easily imported into other systems).

The PR makes it possible to override the Makefile's use of go test to
use any other command tto execute the test. For CI we'll use gotestsum --, where gotestsum expects everything after the -- to be flags for
go test.
We then use environment variables to configure gotestsum (e.g.
GOTESTSUM_JUNITFILE is an env var accepted by gotestsum).

For cri tests, the test suite supports outputing test results to a
directory, these are in junit format already. The file is not named
properly just because the code that creates it (in ginkgo) is not
configured well. We can fix that upstream to give us a better name...
until then I'm keeping those results in a separate dir.

A second workflow is also added so the test results can be summed up and
a report added to the workflow run. The 2nd workflow is required for
this since PR runs do not have access to do some of this due to safety
reasons
(https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)

k8s-ci-robot · 2021-03-25T19:12:58Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

theopenlab-ci · 2021-03-26T23:02:13Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 6m 32s (non-voting)

cpuguy83 · 2021-03-26T23:47:52Z

This should be ready... I don't actually know how test result parser workflow will work out since it won't run until it is merged.

theopenlab-ci · 2021-03-27T00:26:38Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 59s (non-voting)

theopenlab-ci · 2021-03-31T16:56:37Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 48s (non-voting)

This allows us to dig more details out of test runs and maintain a better history. For this we can use `gotestsum`, which is a utility that wraps `go test` so that it outputs test2json (go's format) and output junit (a format more easily imported into other systems). The PR makes it possible to override the Makefile's use of `go test` to use any other command tto executet the test. For CI we'll use `gotestsum --`, where `gotestsum` expects everything after the `--` to be flags for `go test`. We then use environment variables to configure `gotestsum` (e.g. `GOTESTSUM_JUNITFILE` is an env var accepted by `gotestsum`). For cri tests, the test suite supports outputing test results to a directory, these are in junit format already. The file is not named properly just because the code that creates it (in ginkgo) is not configured well. We can fix that upstream to give us a better name... until then I'm keeping those results in a separate dir. A second workflow is also added so the test results can be summed up and a report added to the workflow run. The 2nd workflow is required for this since PR runs do not have access to do some of this due to safety reasons (https://securitylab.github.com/research/github-actions-preventing-pwn-requests/) Signed-off-by: Brian Goff <cpuguy83@gmail.com>

theopenlab-ci · 2021-03-31T17:08:27Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 46s (non-voting)

kzys · 2021-03-31T17:59:52Z

How can I access the result?

cpuguy83 · 2021-03-31T19:23:43Z

Under "checks"

Artifacts:

And then there's the new workflow which should be triggered after the CI workflow runs that would post the results as another check... and optionally a PR comment.

kzys · 2021-04-01T17:39:19Z

+          script: |
+              var fs = require('fs');
+              var path = require('path');
+              var artifacts_path = path.join('${{github.workspace}}', 'artifacts')
+              fs.mkdirSync(artifacts_path, { recursive: true })
+
+              var artifacts = await github.actions.listWorkflowRunArtifacts({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                run_id: ${{ github.event.workflow_run.id }},
+              });
+
+              for (const artifact of artifacts.data.artifacts) {
+                var download = await github.actions.downloadArtifact({
+                    owner: context.repo.owner,
+                    repo: context.repo.repo,
+                    artifact_id: artifact.id,
+                    archive_format: 'zip',
+                });
+                var artifact_path = path.join(artifacts_path, `${artifact.name}.zip`)
+                fs.writeFileSync(artifact_path, Buffer.from(download.data));
+                console.log(`Downloaded ${artifact_path}`);
+              }


Is it possible to have that as a standalone file instead of inlining in YAML?

No, we cannot checkout code in this workflow.

Also to be fair, I have no idea if this will work and no good way to test it here because the workflow won't run until its merged :(

I see. It might be better to package that as our own GitHub actions, in future.

thaJeztah · 2021-04-13T21:52:23Z

@estesp PTAL (as you were working on collecting data from CI)

mikebrow · 2021-04-15T00:25:54Z

+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+
+GO111MODULE=on go install gotest.tools/gotestsum@v1.6.2


nit.. missing new line...

estesp

LGTM

cpuguy83 force-pushed the ci_output_junit branch 5 times, most recently from 0158ab2 to fadb365 Compare March 25, 2021 21:18

cpuguy83 marked this pull request as ready for review March 25, 2021 21:59

cpuguy83 force-pushed the ci_output_junit branch 4 times, most recently from c51aec8 to 096ed25 Compare March 26, 2021 22:54

cpuguy83 force-pushed the ci_output_junit branch from 096ed25 to a36c00f Compare March 27, 2021 00:19

cpuguy83 force-pushed the ci_output_junit branch from a36c00f to 5b45698 Compare March 31, 2021 16:49

cpuguy83 force-pushed the ci_output_junit branch from 5b45698 to af1e2af Compare March 31, 2021 17:00

cpuguy83 requested review from dims, kzys and mxpv and removed request for dims March 31, 2021 17:01

cpuguy83 requested a review from mikebrow March 31, 2021 17:41

kzys reviewed Apr 1, 2021

View reviewed changes

kzys approved these changes Apr 1, 2021

View reviewed changes

dmcgowan approved these changes Apr 1, 2021

View reviewed changes

mikebrow reviewed Apr 15, 2021

View reviewed changes

estesp approved these changes Apr 15, 2021

View reviewed changes

estesp merged commit 6ef0d38 into containerd:master Apr 15, 2021

cpuguy83 deleted the ci_output_junit branch April 15, 2021 19:04

Conversation

cpuguy83 commented Mar 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Mar 25, 2021

Uh oh!

theopenlab-ci Bot commented Mar 26, 2021

Uh oh!

cpuguy83 commented Mar 26, 2021

Uh oh!

theopenlab-ci Bot commented Mar 27, 2021

Uh oh!

theopenlab-ci Bot commented Mar 31, 2021

Uh oh!

theopenlab-ci Bot commented Mar 31, 2021

Uh oh!

kzys commented Mar 31, 2021

Uh oh!

cpuguy83 commented Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kzys Apr 1, 2021

Choose a reason for hiding this comment

Uh oh!

cpuguy83 Apr 1, 2021

Choose a reason for hiding this comment

Uh oh!

cpuguy83 Apr 1, 2021

Choose a reason for hiding this comment

Uh oh!

kzys Apr 1, 2021

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Apr 13, 2021

Uh oh!

mikebrow Apr 15, 2021

Choose a reason for hiding this comment

Uh oh!

estesp left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cpuguy83 commented Mar 25, 2021 •

edited

Loading

cpuguy83 commented Mar 31, 2021 •

edited

Loading