celilo/lunacycle

Fork 0

fix(smoke): repair the three production smoke checks (2 false failures + 1 false pass) #20

Merged

forgejo-admin merged 1 commit from fix-prod-smoke-failures into main

2026-07-31 20:29:24 +00:00

forgejo-admin commented

2026-07-31 20:15:56 +00:00

Owner

The smoke checks ran against production for the first time today. Two failed and one passed. All three were wrong — none of it was a defect in the deployed app.

smoke_ws — could never pass

runWsSmoke's open handler called ws.close() on the line before resolve(). Bun dispatches the close event synchronously from inside close(), and the close listener rejects unconditionally, so reject() settled the promise first on every single run. code=1000, reason='' is exactly what a client-initiated close() with no args produces — the check was reporting its own socket teardown.

Production WebSocket routing was never broken. From both a laptop and celilo-mgr:

curl -i --http1.1 -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
     -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
     https://www.lunacycle.net/api
-> HTTP/1.1 101 Switching Protocols
   Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

Instrumented ordering proof (bun 1.3.3, live prod):

open handler: entered
close handler: code=1000 reason=""     <-- fires inside ws.close()
open handler: ws.close() returned

Fix: resolve before closing. The later reject() then lands on a settled promise and is a no-op.

Worth noting this was never covered anywhere — e2e/tests/deploy.test.ts imports only runSpaSmokeCheck, so runWsSmoke and runApiSmoke had never executed before today.

smoke_spa — could never pass either

Both it and smoke_api authenticated with the smoketest_bot_token hook output. That is an Authentik API token (intent: 'api' — an opaque random key), not an OIDC access token. The server verifies bearers with jwtVerify(token, jwks) and the SPA decodes them with jwtDecode, so both rejected it:

server → ERR_JWS_INVALID → unauthenticated context → UNAUTHORIZED
SPA → decodeJWT throws → clearTokens() → renders "Sign In"

So Header.tsx's {user.name} never rendered and the smoketest_bot marker could not appear. Confirmed by re-running celilo module health lunacycle and correlating the app container's journal — Token verification failed: Invalid Compact JWS at the exact timestamps of both the SPA and API checks.

health_check now performs a real OIDC authorization-code + PKCE login with smoketest_bot_password (already a declared secret), the same flow the SPA's login button drives. This needs no celilo-side changes: the client id and authentik URL are already public in the SPA's own config.js. It adds a smoke_login check and gives the login path production coverage for the first time.

smoke_api — was unfalsifiable

responseMeta in apps/lunacycle-server/src/index.ts hard-codes status: 200 on every tRPC response, errors included:

curl -H 'Authorization: Bearer garbage' '.../api/getActiveMonth?batch=1&input=%7B%7D'
-> HTTP 200
   [{"error":{"message":"Authentication required","data":{"code":"UNAUTHORIZED","httpStatus":401,...

resp.status !== 200 therefore could not fail for any auth or application error. It reported ✓ while the server logged ERR_JWS_INVALID for that same request. Both the API and SPA checks now assert the body carries no tRPC error envelope.

Also

celilo/scripts/health-check.ts never passed artifactDir, so the screenshot + DOM + observed-request post-mortem that spa-smoke.ts carefully writes on failure was discarded in production (e2e passes it). Now written to /tmp/lunacycle-smoke and named in the failure message.

Verification

Against live production:

PKCE mint returns a 3-segment JWT with preferred_username, groups: ["lunacycle-admins"], correct iss
smoke_api, smoke_ws, smoke_spa all pass with that token
the unfixed runWsSmoke reproduced the exact production message; the fixed one passes
runApiSmoke with a bad token now correctly fails with UNAUTHORIZED instead of reporting ✓

Plus 6 new unit tests in tests/unit/smoke-checks.test.ts (WS close-ordering regression, WS genuine-failure still detected, PKCE happy path, PKCE stage-stall, error-envelope detection). Full suite green: 58 vitest + 57 bun tests, lint clean, no new type errors.

The end-to-end verification used lunacycle_admin rather than smoketest_bot, since the bot's password is a sealed module secret. The flow is credential-identical, and the bot can complete it — create_oidc_client's groups argument only creates groups, and the application is policy_engine_mode: 'any' with no policy bindings. Worth confirming on the first real health run after deploy.

Out of scope, flagged for follow-up

responseMeta returns 200 for every error. Breaks HTTP semantics for every client, not just these checks. The smoke fix works around it rather than fixing it.
Production leaks full stack traces in tRPC error bodies.
lunacycle_admin's password is hard-coded as 'lunacycle-rules' in celilo/scripts/setup-web.ts and provisioned into production. A default admin credential in a git repo.
smoketest_bot_token is now unused in-tree. Left in place (it's a legitimate Authentik-API service credential) with a comment so nobody wires it back into app auth.

The smoke checks ran against production for the first time today. Two failed and one passed. **All three were wrong** — none of it was a defect in the deployed app. ## smoke_ws — could never pass `runWsSmoke`'s open handler called `ws.close()` on the line *before* `resolve()`. Bun dispatches the `close` event **synchronously** from inside `close()`, and the close listener rejects unconditionally, so `reject()` settled the promise first on every single run. `code=1000, reason=''` is exactly what a client-initiated `close()` with no args produces — the check was reporting its own socket teardown. Production WebSocket routing was never broken. From both a laptop and celilo-mgr: ``` curl -i --http1.1 -H 'Connection: Upgrade' -H 'Upgrade: websocket' \ -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \ https://www.lunacycle.net/api -> HTTP/1.1 101 Switching Protocols Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo= ``` Instrumented ordering proof (bun 1.3.3, live prod): ``` open handler: entered close handler: code=1000 reason="" <-- fires inside ws.close() open handler: ws.close() returned ``` Fix: resolve before closing. The later `reject()` then lands on a settled promise and is a no-op. Worth noting this was never covered anywhere — `e2e/tests/deploy.test.ts` imports only `runSpaSmokeCheck`, so `runWsSmoke` and `runApiSmoke` had never executed before today. ## smoke_spa — could never pass either Both it and `smoke_api` authenticated with the `smoketest_bot_token` hook output. That is an Authentik **API token** (`intent: 'api'` — an opaque random key), not an OIDC access token. The server verifies bearers with `jwtVerify(token, jwks)` and the SPA decodes them with `jwtDecode`, so both rejected it: - server → `ERR_JWS_INVALID` → unauthenticated context → `UNAUTHORIZED` - SPA → `decodeJWT` throws → `clearTokens()` → renders "Sign In" So `Header.tsx`'s `{user.name}` never rendered and the `smoketest_bot` marker could not appear. Confirmed by re-running `celilo module health lunacycle` and correlating the app container's journal — `Token verification failed: Invalid Compact JWS` at the exact timestamps of both the SPA and API checks. health_check now performs a real **OIDC authorization-code + PKCE login** with `smoketest_bot_password` (already a declared secret), the same flow the SPA's login button drives. This needs no celilo-side changes: the client id and authentik URL are already public in the SPA's own `config.js`. It adds a `smoke_login` check and gives the login path production coverage for the first time. ## smoke_api — was unfalsifiable `responseMeta` in `apps/lunacycle-server/src/index.ts` hard-codes `status: 200` on every tRPC response, errors included: ``` curl -H 'Authorization: Bearer garbage' '.../api/getActiveMonth?batch=1&input=%7B%7D' -> HTTP 200 [{"error":{"message":"Authentication required","data":{"code":"UNAUTHORIZED","httpStatus":401,... ``` `resp.status !== 200` therefore could not fail for any auth or application error. It reported ✓ while the server logged `ERR_JWS_INVALID` for that same request. Both the API and SPA checks now assert the body carries no tRPC error envelope. ## Also `celilo/scripts/health-check.ts` never passed `artifactDir`, so the screenshot + DOM + observed-request post-mortem that `spa-smoke.ts` carefully writes on failure was discarded in production (e2e passes it). Now written to `/tmp/lunacycle-smoke` and named in the failure message. ## Verification Against **live production**: - PKCE mint returns a 3-segment JWT with `preferred_username`, `groups: ["lunacycle-admins"]`, correct `iss` - `smoke_api`, `smoke_ws`, `smoke_spa` all pass with that token - the unfixed `runWsSmoke` reproduced the exact production message; the fixed one passes - `runApiSmoke` with a bad token now correctly fails with `UNAUTHORIZED` instead of reporting ✓ Plus 6 new unit tests in `tests/unit/smoke-checks.test.ts` (WS close-ordering regression, WS genuine-failure still detected, PKCE happy path, PKCE stage-stall, error-envelope detection). Full suite green: 58 vitest + 57 bun tests, lint clean, no new type errors. The end-to-end verification used `lunacycle_admin` rather than `smoketest_bot`, since the bot's password is a sealed module secret. The flow is credential-identical, and the bot can complete it — `create_oidc_client`'s `groups` argument only *creates* groups, and the application is `policy_engine_mode: 'any'` with no policy bindings. Worth confirming on the first real health run after deploy. ## Out of scope, flagged for follow-up 1. **`responseMeta` returns 200 for every error.** Breaks HTTP semantics for every client, not just these checks. The smoke fix works around it rather than fixing it. 2. **Production leaks full stack traces** in tRPC error bodies. 3. **`lunacycle_admin`'s password is hard-coded** as `'lunacycle-rules'` in `celilo/scripts/setup-web.ts` and provisioned into production. A default admin credential in a git repo. 4. `smoketest_bot_token` is now unused in-tree. Left in place (it's a legitimate Authentik-API service credential) with a comment so nobody wires it back into app auth.

forgejo-admin added 1 commit

2026-07-31 20:15:56 +00:00

fix(smoke): repair the three production smoke checks

pr-validate / validate (pull_request) Successful in 15s

Details

release / version (pull_request) Successful in 9s

Details

release / e2e (pull_request) Successful in 2m21s

Details

release / publish (pull_request) Has been skipped

Details

ea0dca3c7e

They ran against production for the first time today and reported two
false failures and one false pass. None of it was a defect in the
deployed app.

smoke_ws could never pass on bun. The open handler called ws.close()
before resolve(), and bun dispatches 'close' synchronously, so the close
listener's unconditional reject() won the race every run — reporting
"closed before open (code=1000)" against an endpoint that had just
completed the upgrade. curl --http1.1 against prod returns 101 Switching
Protocols; Caddy's websocket routing was never broken. Resolve first.

smoke_spa could never pass either. It and smoke_api authenticated with
the smoketest_bot_token hook output, which is an authentik API token
(intent: 'api') — an opaque key, not a signed JWT. The server's
jwtVerify rejected it with ERR_JWS_INVALID and the SPA's jwtDecode threw
and cleared the session, so the app rendered "Sign In" and the
smoketest_bot marker never appeared. health_check now performs a real
OIDC authorization-code + PKCE login with smoketest_bot_password, the
same flow the SPA's login button drives, yielding a genuine token with
preferred_username and groups claims. That adds a smoke_login check and
makes the login path itself covered in production for the first time.

smoke_api was unfalsifiable. responseMeta hard-codes status: 200 on
every tRPC response including errors, so asserting status === 200 passed
while the server was rejecting the very same request. Both the API and
SPA checks now assert the body carries no tRPC error envelope.

Also wires artifactDir into the production SPA check — e2e passed it,
prod didn't, so the screenshot/DOM/request post-mortem was discarded
exactly when it was needed.

Verified against live production: the PKCE mint returns a 3-segment JWT
with the expected claims, and all three checks pass with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

forgejo-admin merged commit 25d52a2db5 into main

2026-07-31 20:29:24 +00:00

forgejo-admin referenced this pull request from a commit

2026-07-31 20:29:25 +00:00

Merge pull request 'fix(smoke): repair the three production smoke checks (2 false failures + 1 false pass)' (#20) from fix-prod-smoke-failures into main

forgejo-admin referenced this pull request

2026-07-31 20:38:07 +00:00

tRPC server returns HTTP 200 for every error, including UNAUTHORIZED #22

forgejo-admin referenced this pull request

2026-07-31 20:38:20 +00:00

Production leaks server stack traces in tRPC error responses #23

forgejo-admin referenced this pull request