fix(smoke): mint a real OIDC token in the post-deploy smoke handler #28

Merged

forgejo-admin merged 1 commit from fix-smoke-handler-token into main

2026-07-31 23:58:07 +00:00

forgejo-admin commented

2026-07-31 21:49:31 +00:00

Owner

The smoke-after-deploy bus subscriber has never once succeeded. The deliveries table on celilo-mgr shows 3 failed, 0 succeeded, and it has been that way since it was wired.

Surfaced by the celilo-side poll-CD investigation, which swept every subscriber's delivery outcomes and found this one alongside the poll bug.

Same root cause as #20 — and I missed this caller

smoke-handler.ts authenticated with the smoketest_bot_token secret: an authentik API token (intent: 'api'), an opaque key rather than a signed JWT. The server verifies bearers with jwtVerify and the SPA decodes them with jwtDecode, so both reject it and the SPA never renders the smoketest_bot marker.

That is exactly what #20 fixed for the health_check hook, in the same three functions. I missed it because I scoped my grep to celilo/scripts/health-check.ts instead of celilo/scripts/, and fixed only the path the bug report named. This is the "grep every caller before you edit" rule, and I didn't follow it.

Re-grepped the whole repo this time. The complete caller set is:

caller	status
`celilo/scripts/health-check.ts`	fixed in #20
`celilo/scripts/smoke-handler.ts`	fixed here
`e2e/tests/deploy.test.ts`	always correct — derives its own PKCE token

No others.

The fix

Reads smoketest_bot_password and calls mintAccessToken, mirroring health_check — including deriving the authentik origin from idp_base_url with the same auth.<domain> fallback. Also passes artifactDir, so a failure here leaves the same screenshot / DOM / request post-mortem in /tmp/lunacycle-smoke that the hook path leaves.

New evidence for the #20 diagnosis

celilo module secret get does exist, and on celilo-mgr it returns a 60-character dotless string for smoketest_bot_token. A JWT has three dot-separated segments. That is the last piece of direct evidence that the token was never a valid bearer for this app — #20 inferred it from ERR_JWS_INVALID in the server log, and this confirms it from the other end.

Note on why it stayed invisible

The delivery status is failed rather than abandoned, meaning the handler ran and reported failure itself. Nothing surfaces that unless you query the deliveries table — the same invisible-failure shape that hid the poll-CD bug for weeks. Worth a monitor on subscribers with a zero success rate; that's celilo-side, not filed here.

Verified

Typecheck clean (only the pre-existing dual-playwright-core Browser mismatch, which also affects health-check.ts on main), lint clean, 58 vitest + 57 bun tests green.

Not verified end-to-end: that needs a deploy to fire deploy.completed.lunacycle. I also could not test mintAccessToken with the real smoketest_bot credentials — reading that secret into an env var was blocked, correctly. So #20's caveat still stands: the first post-deploy run is what proves the bot can complete the OIDC flow. If it can't, smoke_login in health_check and the mint step here both name the exact failing hop.

The `smoke-after-deploy` bus subscriber has **never once succeeded**. The deliveries table on celilo-mgr shows `3 failed, 0 succeeded`, and it has been that way since it was wired. Surfaced by the celilo-side poll-CD investigation, which swept every subscriber's delivery outcomes and found this one alongside the poll bug. ## Same root cause as #20 — and I missed this caller `smoke-handler.ts` authenticated with the `smoketest_bot_token` secret: an authentik API token (`intent: 'api'`), an opaque key rather than a signed JWT. The server verifies bearers with `jwtVerify` and the SPA decodes them with `jwtDecode`, so both reject it and the SPA never renders the `smoketest_bot` marker. That is exactly what #20 fixed for the `health_check` hook, in the same three functions. **I missed it because I scoped my grep to `celilo/scripts/health-check.ts` instead of `celilo/scripts/`**, and fixed only the path the bug report named. This is the "grep every caller before you edit" rule, and I didn't follow it. Re-grepped the whole repo this time. The complete caller set is: | caller | status | |---|---| | `celilo/scripts/health-check.ts` | fixed in #20 | | `celilo/scripts/smoke-handler.ts` | **fixed here** | | `e2e/tests/deploy.test.ts` | always correct — derives its own PKCE token | No others. ## The fix Reads `smoketest_bot_password` and calls `mintAccessToken`, mirroring `health_check` — including deriving the authentik origin from `idp_base_url` with the same `auth.<domain>` fallback. Also passes `artifactDir`, so a failure here leaves the same screenshot / DOM / request post-mortem in `/tmp/lunacycle-smoke` that the hook path leaves. ## New evidence for the #20 diagnosis `celilo module secret get` does exist, and on celilo-mgr it returns a **60-character dotless string** for `smoketest_bot_token`. A JWT has three dot-separated segments. That is the last piece of direct evidence that the token was never a valid bearer for this app — #20 inferred it from `ERR_JWS_INVALID` in the server log, and this confirms it from the other end. ## Note on why it stayed invisible The delivery status is `failed` rather than `abandoned`, meaning the handler ran and reported failure itself. Nothing surfaces that unless you query the deliveries table — the same invisible-failure shape that hid the poll-CD bug for weeks. Worth a monitor on subscribers with a zero success rate; that's celilo-side, not filed here. ## Verified Typecheck clean (only the pre-existing dual-`playwright-core` `Browser` mismatch, which also affects `health-check.ts` on `main`), lint clean, 58 vitest + 57 bun tests green. **Not** verified end-to-end: that needs a deploy to fire `deploy.completed.lunacycle`. I also could not test `mintAccessToken` with the real `smoketest_bot` credentials — reading that secret into an env var was blocked, correctly. So #20's caveat still stands: the first post-deploy run is what proves the bot can complete the OIDC flow. If it can't, `smoke_login` in health_check and the mint step here both name the exact failing hop.

forgejo-admin added 1 commit

2026-07-31 21:49:31 +00:00

fix(smoke): mint a real OIDC token in the post-deploy smoke handler

pr-validate / validate (pull_request) Successful in 15s

Details

89b2dd9016

The `smoke-after-deploy` bus subscriber has never once succeeded — the
deliveries table on celilo-mgr shows 3 failed, 0 succeeded, and it has been
that way since it was wired.

Same root cause as the health_check hook fixed in #20, in the same three
functions: it authenticated with the `smoketest_bot_token` secret, an authentik
API token (`intent: 'api'`) that is an opaque key rather than a signed JWT. The
server verifies bearers with jwtVerify and the SPA decodes them with jwtDecode,
so both reject it and the SPA never renders the smoketest_bot marker.

I missed this caller in #20 because I scoped the grep to health-check.ts instead
of celilo/scripts/, and fixed only the path the report named. Re-grepped the
whole repo this time: health_check, this handler, and e2e are the complete set
of callers, and e2e derives its own PKCE token so it was always correct.

The handler now reads smoketest_bot_password and calls mintAccessToken, exactly
as health_check does, deriving the authentik origin from idp_base_url with the
same `auth.<domain>` fallback. Also passes artifactDir so a failure leaves the
same post-mortem the hook path leaves.

Confirmed on celilo-mgr that `celilo module secret get lunacycle
smoketest_bot_token` returns a 60-character dotless string — not a JWT, which is
the last piece of direct evidence for the #20 diagnosis.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

forgejo-admin merged commit 79662034f0 into main

2026-07-31 23:58:07 +00:00

forgejo-admin referenced this pull request from a commit

2026-07-31 23:58:08 +00:00

Merge pull request 'fix(smoke): mint a real OIDC token in the post-deploy smoke handler' (#28) from fix-smoke-handler-token into main