Skip to content

test(ui): add UI auth E2E tests - #7167

Open
luizhf42 wants to merge 3 commits into
masterfrom
test/ui/auth-e2e
Open

luizhf42 wants to merge 3 commits into
masterfrom
test/ui/auth-e2e

Conversation

@luizhf42

@luizhf42 luizhf42 commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

What

Adds the first Playwright E2E suite covering the authentication domain: sign-in, invalid credentials, sign-out, account lockout with recovery, invitation acceptance (logged-in, not-logged-in, new-user sign-up), and namespace switching.

Why

The E2E infrastructure landed but had only 3 starter tests. This fills out the auth domain with real user flows that catch wiring issues between the UI and the API: wrong button roles, missing redirects, lockout timing, invitation redirect loops.

Changes

  • auth.spec.ts: 8 tests across 4 describe blocks. Fixture users and namespaces are created per run via the stack CLI with randomized names to avoid collisions on a reused stack
  • helpers.ts: shared signIn, dismissWizard, createUser, createNamespace helpers. Uses networkidle instead of a fixed sleep for wizard detection
  • api.ts: typed API helpers (login, createInvitationLink) using Playwright's APIRequestContext for test fixture creation
  • env.ts: added edition awareness (isCommunity, isEnterprise, isCloud) for future edition-gated tests
  • OpenAPI: marked link as required in the generateInvitationLink 200 response (the generated type had it optional)

Testing

npm run e2e:test -w @shellhub/console

@luizhf42 luizhf42 self-assigned this Sep 22, 2026
@luizhf42
luizhf42 added this pull request to stack #7168 September 22, 2026 14:48
@luizhf42
luizhf42 requested review from a team as code owners September 22, 2026 14:48
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Claude finished @luizhf42's task in 14m 41s —— View job


Code Review Complete

Reviewed 5 files. 8 inline issues posted. Standards: 0 violations, 1 judgement call. Spec: did not run. Correctness: 7.

Standards

No hard violations. One judgement call: the edition flags in ui/apps/console/e2e/env.ts:16-24 have no consumer yet, and the as cast means a typo in E2E_EDITION becomes a skip that never fires.

Correctness

test.beforeAll(async ({ request }) => …) at ui/apps/console/e2e/auth.spec.ts:75 asks for a test-scoped fixture in a worker-scoped hook — Playwright throws before any test body runs, so all three accept invitation tests fail on every run.

Also posted: dismissWizard's non-retrying isVisible() leaves the wizard's modal overlay blocking the next click (helpers.ts:13); new URL("") throws under the enterprise/cloud matrix legs, where GenerateInvitationLink returns an empty link for an account that already exists (auth.spec.ts:98); createNamespace's "already exists" filter misses Community's single-namespace error after any server restart (helpers.ts:39); the namespace-switch test passes vacuously in one ordering (auth.spec.ts:179) and leaves admin's preferred namespace moved for every later spec (auth.spec.ts:167); Omit on a string union constrains nothing (api.ts:54).

Spec

The Spec axis did not run: the PR and its commits reference no issue, and a spec is never reconstructed from the PR title.

Additional notes

The OpenAPI change is sound — server/api/routes/invitation.go always emits the link key, required matches the wire contract, and ui/apps/console/src/client/ is generated at build time, so there is no committed artifact to drift. cloud/ generates from the same spec the same way.

The waitForTimeout(2000) at auth.spec.ts:62 came up and was dropped: its reason (clock skew between the 429 epoch the countdown derives from and the server-side lockout TTL) is recorded in the commit body, which is where code-style.md puts it.


If you've addressed the feedback and want a new review, tag @shellhub-io/admin and a team member can trigger it.
· test/ui/auth-e2e

Comment thread ui/apps/console/e2e/auth.spec.ts
Comment thread ui/apps/console/e2e/helpers.ts
Comment thread ui/apps/console/e2e/auth.spec.ts
Comment thread ui/apps/console/e2e/helpers.ts Outdated
Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/e2e/api.ts Outdated
Comment thread ui/apps/console/e2e/env.ts
@luizhf42
luizhf42 force-pushed the test/ui/auth-e2e branch 2 times, most recently from 77b402a to 70405ef Compare September 22, 2026 19:47
@luizhf42
luizhf42 requested a review from a team as a code owner September 22, 2026 19:47
@luizhf42
luizhf42 removed the request for review from a team September 22, 2026 19:53
@gustavosbarreto

Copy link
Copy Markdown
Member

/review

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Claude finished @gustavosbarreto's task in 18m 16s —— View job


Code Review Complete

Reviewed 9 files. 8 inline issues posted. Standards: 0 violations, 3 judgement calls. Spec: did not run. Correctness: 5.

Standards

No hard violations. The previous round's Omit → Exclude fix is correct, and the rewritten wizard unit tests lose no coverage — the dropped step-1-vs-final-step rows pinned a branch the diff deleted. Three judgement calls: the aria-label guard at auth.spec.ts:189 still passes for the wrong namespace, the doc comments at WelcomeWizard.tsx:29-32 omit the manual-add link's new dismissal path, and api.ts:2-6 reaches past the @/client barrel into types.gen.

Correctness

Two tests are silently skipped in CI right now. The enterprise leg of the current head's run reports 2 skipped, 6 passed — the null-sig guard added last round (auth.spec.ts:106-109) converted the new URL("") crash into a green skip, and on enterprise GenerateInvitationLink returns "" for every existing user by construction, so both tests can never obtain a link. env.ts's isEnterprise flag is the fix and its first real consumer.

Also posted: the onClose/onDismiss collapse makes Escape or a stray backdrop click permanently suppress onboarding with no way to bring it back (WelcomeWizard.tsx:78); the widened multi-tenancy catch swallows Community's single-namespace constraint, making that leg a false green (helpers.ts:44); the namespace switch-back can't fail and doesn't run on the failure path, leaving the shared admin's preferred namespace moved (auth.spec.ts:200-202); serverAdmin builds a shell string, so any fixture password with a space or ; breaks (helpers.ts:21-24).

Spec

The Spec axis did not run: neither the PR nor its three commits reference an issue, and a spec is never reconstructed from the PR title.

Additional notes

The declined beforeAll({ request }) finding from the previous round was wrong, and the author's reply was right — all three accept invitation tests execute and pass on community and cloud. It is withdrawn, not carried forward. No cap dropped any finding.

One verified-clean item worth recording: the OpenAPI change is honest about the wire contract — server/api/routes/invitation.go:90 always emits the link key and cloud/ adds no override — but the value is "" on the direct-membership path, so the now-non-optional link: string hides the empty case from callers doing new URL(link). That is exactly the case the enterprise skip is papering over.


If you've addressed the feedback and want a new review, tag @shellhub-io/admin and a team member can trigger it.
· test/ui/auth-e2e

Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/src/components/wizard/WelcomeWizard.tsx
Comment thread ui/apps/console/e2e/helpers.ts Outdated
Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/src/components/wizard/WelcomeWizard.tsx
Comment thread ui/apps/console/e2e/helpers.ts Outdated
Comment thread ui/apps/console/e2e/api.ts Outdated
@gustavosbarreto

Copy link
Copy Markdown
Member

@luizhf42 about the waitForTimeout(2000) in the lockout test: the cause wasn't clock skew. The browser and the containers share the host clock.

The server announced the deadline in X-Account-Lockout truncated to whole seconds, while the account-lockout= key in Redis lived the full TTL, and HasAccountLockout blocked while the key existed. So the login kept getting a 429 for up to 1s after the countdown finished. I measured it on a dev stack: first 200 at deadline +0.699s.

#7181 fixed it (merged, closes #7179): the server now unlocks at exactly the announced second. Once this branch picks it up, you can:

  • drop the waitForTimeout(2000) and log in right after "Your timeout has finished"
  • remove the clock skew paragraph from the test(ui): add auth domain e2e tests commit message

This PR is based on test/ui/playwright (#7080), so the fix only arrives here after that branch is rebased onto master.

@luizhf42

Copy link
Copy Markdown
Member Author

@luizhf42 about the waitForTimeout(2000) in the lockout test: the cause wasn't clock skew. The browser and the containers share the host clock.

The server announced the deadline in X-Account-Lockout truncated to whole seconds, while the account-lockout= key in Redis lived the full TTL, and HasAccountLockout blocked while the key existed. So the login kept getting a 429 for up to 1s after the countdown finished. I measured it on a dev stack: first 200 at deadline +0.699s.

#7181 fixed it (merged, closes #7179): the server now unlocks at exactly the announced second. Once this branch picks it up, you can:

  • drop the waitForTimeout(2000) and log in right after "Your timeout has finished"
  • remove the clock skew paragraph from the test(ui): add auth domain e2e tests commit message

This PR is based on test/ui/playwright (#7080), so the fix only arrives here after that branch is rebased onto master.

Done. I rebased both branches, removed the 2s timeout, and reworded the commit body.

@gustavosbarreto

Copy link
Copy Markdown
Member

/review

Base automatically changed from test/ui/playwright to master September 24, 2026 14:06
@luizhf42
luizhf42 force-pushed the test/ui/auth-e2e branch 4 times, most recently from 337535c to f3ead88 Compare September 25, 2026 12:03
@gustavosbarreto

Copy link
Copy Markdown
Member

/review

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Claude finished @gustavosbarreto's task in 18m 56s —— View job


Code Review

  • Gathered PR context
  • Code review complete (Standards, Correctness)
  • Posted inline comments

Third round. Everything raised in rounds 1 and 2 is genuinely addressed at f3ead88 — dismissWizard now uses waitFor rather than isVisible(), the empty-link case throws, serverAdmin runs execFileSync with an argv array so neither E2E_STACK_NAME nor a generated username can inject, and the isEnterprise skip carries a stated reason and is correctly scoped (EnableDirectMembership() registers only under envs.Enterprise, so Cloud and Community both return a link). CI is green on all 11 checks; the UI E2E matrix runs 8 tests per edition with the two invitation tests skipped on enterprise, as intended.

8 findings posted: 2 correctness, 2 hard violations of code-style.md, 4 judgement calls.

The two worth reading first are both in namespace switching. The describe gives the admin a second namespace, which Community forbids — it passes only because reconcileInstanceBinding never re-runs after admin user create sets system.Setup, so a server restart at any point after stack up turns the whole describe (plus the lockout and invitee fixtures) red. And the precondition at line 190 uses toContainText("e2e"), which e2e-switch-<hex> also satisfies, so the assertion that exists to separate the two namespaces can't.

Dropped before posting

  • The env.ts speculative-generality and unchecked-as-cast pair — round 1 raised both and you declined; no new argument.
  • required: [link] versus the empty string the Enterprise direct-membership path returns — round 2's closing notes already covered it.
  • The namespace switch-back — round 2, and you dropped it deliberately once stack reuse went away. The only new argument was a hypothetical second spec file.
  • A claim that auth.spec.ts:103's if (!link) throw guards became vestigial once link was marked required. Validation killed it: required forbids the key being absent, not the value being "", and GenerateInvitationLink returns ("", nil) on the direct-membership path — those guards are the only thing catching it.
  • The 75s lockout timeout as a standalone correctness finding. It's real but conditional on setup exceeding ~15s of the 90s cap, which validation put below the bar; the point is folded into the code-style.md comment at line 48 instead.

The Spec axis did not run. Neither the PR body nor any of its three commits (f3ead88, 26eb9f0, 2fb57d0) references an issue, and a spec is never reconstructed from the PR title.

If you've addressed the feedback and want a new review, tag @shellhub-io/admin and a team member can trigger it.
· test/ui/auth-e2e

Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/e2e/auth.spec.ts
Comment thread ui/apps/console/e2e/helpers.ts
Comment thread ui/apps/console/src/components/wizard/WelcomeWizard.tsx
Comment thread ui/apps/console/e2e/auth.spec.ts Outdated
Comment thread ui/apps/console/e2e/helpers.ts
Comment thread ui/apps/console/e2e/api.ts
The `generateInvitationLink` 200 response declared the `link` property
without listing it in `required`, so the generated TypeScript type made
it optional. Callers had to assert `link!` even though the server always
returns it.
handleClose routed to onClose at step 1, which set React state but
never wrote to localStorage. After a namespace switch (full page
reload), the component remounted, found no flag, and showed the wizard
again for the same tenant.

Remove the onClose/onDismiss distinction and pass onDismiss directly
to BaseDialog and the X button so any dismissal calls markWelcomeSeen.
Cover sign-in, invalid credentials, sign-out, account lockout (trigger +
recovery) and invitation acceptance (logged-in, not-logged-in, new-user
sign-up).

Fixture users and namespaces are created through the stack CLI
(`docker compose exec server /server admin`) with randomized names, so
each test gets its own fixtures.

The two existing-user invitation tests skip on enterprise. Enterprise
enables direct membership, so inviting a user who already has an
account adds them to the namespace and returns no invitation link.
Community and cloud return the link and run both tests.

The lockout test fails three logins because `StoreLoginAttempt`
(`pkg/cache/cache_redis.go`) starts locking on the third attempt, for
`4^(attempt-3)` minutes, so the third failure locks the account for one
minute. The countdown wait allows 75s to cover that minute plus page
and network overhead.

`dismissWizard` treats a wizard that never appears within 5s as absent
and carries on. Callers use it after every sign-in, and the wizard is
legitimately missing once the tenant dismissed it or the namespace has
an accepted device. Tests that check the wizard itself live in the
`WelcomeWizard` unit tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants