caddy's reconcile_routes crashes with "{} is not iterable", so every public_web route change is abandoned and static-site consumers cannot deploy #1401

New issue

Open

opened 2026-09-21 06:23:07 +00:00 by forgejo-admin · 1 comment

forgejo-admin commented

2026-09-21 06:23:07 +00:00

Owner

caddy.reconcile-web-routes is the only subscriber to public_web.routes_changed.
Its handler exits 1 with {} is not iterable and is retried 3 times and then
abandoned, so the route change is never applied. The consumer's deploy then
fails with a message that describes a timeout, not a crash:

public_web reconcile for <module> (route change) did not finish within the
deadline (0 ok, 0 failed of 1 change event(s)) — caddy did not confirm the change is live.

0 ok, 0 failed of 1 is the tell. The change was never counted, because the
handler died before reporting either outcome.

Evidence

celilo events list-failed on celilo-mgr, 2026-09-21. Ten deliveries, every one
status: abandoned, attempts: 3, all on public_web.routes_changed:

eventId 73380  public_web.routes_changed  caddy.reconcile-web-routes  abandoned  attempts 3
  error: handler exited with code 1
  --- handler output (last 1000 chars) ---
  [caddy:reconcile_routes] ✗ Hook reconcile_routes failed: {} is not iterable
  ✗ Error: {} is not iterable

Identical for eventIds 73337, 73338, 73339, 73340, 73341, 73342, 73379, 73380,
and 55594 (an older instance, so this is not new tonight).

Likely cause

modules/caddy/scripts/reconcile-routes.ts:

const routes = (await capabilities.web_routes?.getAllRoutes()) ?? [];

?? guards null/undefined only. If getAllRoutes() resolves to {} — an
empty object rather than an empty array — the coalesce does not fire, {}
flows into convergeCaddy, and the first iteration over it throws exactly
{} is not iterable.

I read that line in a worktree, not on the deployed caddy@3.0.0+2, so confirm
the deployed copy before fixing. The runtime error is not in doubt; the specific
line is my inference from it.

Worth establishing why getAllRoutes() yields {}: an empty-result
serialisation across the capability boundary is the obvious candidate, and if so
the same shape may affect other capability getters, which is a wider fix than
hardening this one call site.

Impact — this is not a corner case

It blocks every static-site consumer from deploying. Observed tonight:

bna-yard-sale — a new install. Public DNS registered fine
(bna-yard-sale.org and www. → 75.164.228.76), but the site block and cert
never appear. Module stuck in DEPLOYING across four attempts.
celilo-website — has been failing its 15-minute registry poll repeatedly
with this exact deadline message, burning a +N build revision on each
attempt (1.0.5+2 → +3 → +4 → +5 → +6 → +7 → +8 → 1.0.6+1 all visible in the
failed-delivery log). That is days of silent failure attributed to a timeout.

Reverse-proxy registrations are not affected: forgejo
(registerReverseProxy) and celilo-apt-repo (register_route) both succeeded
tonight through the same subscriber. So the crash appears specific to the route
shape produced by publishStaticSite. Confirm that split — it is the fastest
route to a minimal reproduction.

Scope

modules/caddy/scripts/reconcile-routes.ts, the getAllRoutes() coalesce.
convergeCaddy / reconcileCaddyfile — whichever iteration actually throws.
web_routes.getAllRoutes(): establish what it returns when there are no rows,
or when the caller is the provider itself, and whether {} is ever a legal
result.
The consumer-facing error text, which reports a deadline for what is a crash.

Acceptance

A minimal reproduction: a publishStaticSite route change that makes
reconcile_routes throw, before any fix.
reconcile_routes does not crash on any result getAllRoutes() can
legally return. If {} is not legal, fix it at the source rather than
only defending at the call site.
A consumer whose route change is abandoned gets an error naming the
handler failure, not "did not finish within the deadline". A crash and a
timeout must not be indistinguishable to the operator — that is what hid
this for days.
celilo-website upgrades cleanly through the registry poll without
burning a build revision per attempt.
Recurrence gate: a test covering getAllRoutes() returning an empty
non-array. Mutation-test it — revert the fix and watch it fail.

celilo#1399 — the dispatcher liveness defect. Distinct, and I conflated them
when filing that one. #1399's no dispatcher running — not waiting is real
and separately measured. But the deadline (0 ok, 0 failed of N) failures I
attributed to dispatcher lag there are this bug. Two defects, one visible
symptom. Correcting #1399 accordingly.
celilo#1361 — publishStaticSite … sourceDir. Different failure, also on
static-site consumers; those fail earlier, at request validation, and never
reach the reconcile.

Measured 2026-09-21 during the fleet-readiness sweep (ce-42q4.3).

`caddy.reconcile-web-routes` is the only subscriber to `public_web.routes_changed`. Its handler exits 1 with `{} is not iterable` and is retried 3 times and then **abandoned**, so the route change is never applied. The consumer's deploy then fails with a message that describes a *timeout*, not a crash: ``` public_web reconcile for <module> (route change) did not finish within the deadline (0 ok, 0 failed of 1 change event(s)) — caddy did not confirm the change is live. ``` `0 ok, 0 failed of 1` is the tell. The change was never *counted*, because the handler died before reporting either outcome. ## Evidence `celilo events list-failed` on celilo-mgr, 2026-09-21. Ten deliveries, every one `status: abandoned`, `attempts: 3`, all on `public_web.routes_changed`: ``` eventId 73380 public_web.routes_changed caddy.reconcile-web-routes abandoned attempts 3 error: handler exited with code 1 --- handler output (last 1000 chars) --- [caddy:reconcile_routes] ✗ Hook reconcile_routes failed: {} is not iterable ✗ Error: {} is not iterable ``` Identical for eventIds 73337, 73338, 73339, 73340, 73341, 73342, 73379, 73380, and 55594 (an older instance, so this is not new tonight). ## Likely cause `modules/caddy/scripts/reconcile-routes.ts`: ```ts const routes = (await capabilities.web_routes?.getAllRoutes()) ?? []; ``` `??` guards `null`/`undefined` only. If `getAllRoutes()` resolves to `{}` — an empty **object** rather than an empty array — the coalesce does not fire, `{}` flows into `convergeCaddy`, and the first iteration over it throws exactly `{} is not iterable`. I read that line in a worktree, not on the deployed `caddy@3.0.0+2`, so confirm the deployed copy before fixing. The runtime error is not in doubt; the specific line is my inference from it. Worth establishing *why* `getAllRoutes()` yields `{}`: an empty-result serialisation across the capability boundary is the obvious candidate, and if so the same shape may affect other capability getters, which is a wider fix than hardening this one call site. ## Impact — this is not a corner case **It blocks every static-site consumer from deploying.** Observed tonight: - `bna-yard-sale` — a new install. Public DNS registered fine (`bna-yard-sale.org` and `www.` → 75.164.228.76), but the site block and cert never appear. Module stuck in `DEPLOYING` across four attempts. - `celilo-website` — has been failing its 15-minute registry poll repeatedly with this exact deadline message, burning a `+N` build revision on each attempt (1.0.5+2 → +3 → +4 → +5 → +6 → +7 → +8 → 1.0.6+1 all visible in the failed-delivery log). That is days of silent failure attributed to a timeout. Reverse-proxy registrations are **not** affected: `forgejo` (`registerReverseProxy`) and `celilo-apt-repo` (`register_route`) both succeeded tonight through the same subscriber. So the crash appears specific to the route shape produced by `publishStaticSite`. Confirm that split — it is the fastest route to a minimal reproduction. ## Scope - `modules/caddy/scripts/reconcile-routes.ts`, the `getAllRoutes()` coalesce. - `convergeCaddy` / `reconcileCaddyfile` — whichever iteration actually throws. - `web_routes.getAllRoutes()`: establish what it returns when there are no rows, or when the caller is the provider itself, and whether `{}` is ever a legal result. - The consumer-facing error text, which reports a deadline for what is a crash. ## Acceptance - [ ] A minimal reproduction: a `publishStaticSite` route change that makes `reconcile_routes` throw, before any fix. - [ ] `reconcile_routes` does not crash on any result `getAllRoutes()` can legally return. If `{}` is not legal, fix it at the source rather than only defending at the call site. - [ ] A consumer whose route change is abandoned gets an error naming the handler failure, not "did not finish within the deadline". A crash and a timeout must not be indistinguishable to the operator — that is what hid this for days. - [ ] `celilo-website` upgrades cleanly through the registry poll without burning a build revision per attempt. - [ ] Recurrence gate: a test covering `getAllRoutes()` returning an empty non-array. Mutation-test it — revert the fix and watch it fail. ## Related - celilo#1399 — the dispatcher liveness defect. **Distinct, and I conflated them when filing that one.** #1399's `no dispatcher running — not waiting` is real and separately measured. But the `deadline (0 ok, 0 failed of N)` failures I attributed to dispatcher lag there are *this* bug. Two defects, one visible symptom. Correcting #1399 accordingly. - celilo#1361 — `publishStaticSite … sourceDir`. Different failure, also on static-site consumers; those fail earlier, at request validation, and never reach the reconcile. Measured 2026-09-21 during the fleet-readiness sweep (ce-42q4.3).

forgejo-admin commented

2026-09-21 06:38:12 +00:00

Author

Owner

The crash does not reproduce on the deployed code

Measured on celilo-mgr, 2026-09-21 06:25–06:31Z, against the deployed caddy@3.0.0+2:

celilo module run-hook caddy reconcile_routes --debug → ✓ Caddy converged: 11 hostname(s), 16 route(s), 4 site(s), 5 companion redirect(s), 2 module(s) recorded in ERROR. Clean, no crash.
Two public_web.routes_changed events have been delivered since the dispatcher restarted at 05:46Z — 73821 (triggeredBy: bna-yard-sale, 05:51Z) and 73873 (a probe I emitted, 06:29Z). Neither failed.
celilo events list-failed --subscriber caddy.reconcile-web-routes still shows the same 10, latestFinishedAt 00:52Z. Nothing new.

So the ten abandoned deliveries all predate 05:46Z and something between 00:52Z and 05:51Z stopped it. I could not establish what. Worth ruling out first: the subscriber's handler is celilo events run-hook caddy reconcile-web-routes, a fresh subprocess against the on-disk binary each tick, so a stale long-running dispatcher is not the mechanism — that was my first theory and it is wrong.

The {} source is therefore unestablished, and the branch below does not claim to fix it.

What the branch does fix

Branch polecat/ce-yc9n (bead ce-yc9n).

Acceptance item 3 — a crash must not read as a timeout. Root-caused, and it is in bus.markFailed: a retried failure is written status='pending' with last_error set (packages/event-bus/src/bus.ts:532). waitForRouteReconcile counts only settled deliveries, so the deadline expires with 0 ok, 0 failed of 1 while the handler's own words sit unread in the row. RouteReconcileWaitResult now carries lastError, and both the timeout and failed-delivery errors quote it. Test: carries the crashing handler error out of a timeout.

Acceptance item 5 — recurrence gate. requireRouteTable in modules/caddy/scripts/reconcile.ts, used by reconcile-routes.ts and setup-network.ts. It throws on a non-array rather than coercing to [] — an empty route table is a legitimate state that renders the holding page, so coercing would converge caddy into serving nothing and drop every site block, silently. The message names the capability and the shape, which is what turns the next occurrence into an obtainable minimal repro (item 1).

Both gates were watched failing under mutation, not reasoned about.

Items 1, 2 and 4 are NOT met and should stay open: no minimal repro, no source-level fix for {}, and celilo-website's registry poll is unverified across a full cycle.

Gates: biome exit 0 (Checked 1930 files), check:typecheck 27/27, check:modules 0 fail, celilo-events 21 pass, caddy scripts 19 pass. caddy bumped to 3.0.1; @celilo/cli changeset added.

## The crash does not reproduce on the deployed code Measured on celilo-mgr, 2026-09-21 06:25–06:31Z, against the deployed `caddy@3.0.0+2`: - `celilo module run-hook caddy reconcile_routes --debug` → `✓ Caddy converged: 11 hostname(s), 16 route(s), 4 site(s), 5 companion redirect(s), 2 module(s) recorded in ERROR`. Clean, no crash. - Two `public_web.routes_changed` events have been delivered since the dispatcher restarted at 05:46Z — **73821** (`triggeredBy: bna-yard-sale`, 05:51Z) and **73873** (a probe I emitted, 06:29Z). Neither failed. - `celilo events list-failed --subscriber caddy.reconcile-web-routes` still shows the same **10**, `latestFinishedAt` 00:52Z. Nothing new. So the ten abandoned deliveries all predate 05:46Z and something between 00:52Z and 05:51Z stopped it. I could not establish what. Worth ruling out first: the subscriber's handler is `celilo events run-hook caddy reconcile-web-routes`, a **fresh subprocess against the on-disk binary each tick**, so a stale long-running dispatcher is *not* the mechanism — that was my first theory and it is wrong. **The `{}` source is therefore unestablished, and the branch below does not claim to fix it.** ## What the branch does fix Branch `polecat/ce-yc9n` (bead `ce-yc9n`). **Acceptance item 3 — a crash must not read as a timeout.** Root-caused, and it is in `bus.markFailed`: a retried failure is written `status='pending'` with `last_error` set (`packages/event-bus/src/bus.ts:532`). `waitForRouteReconcile` counts only *settled* deliveries, so the deadline expires with `0 ok, 0 failed of 1` while the handler's own words sit unread in the row. `RouteReconcileWaitResult` now carries `lastError`, and both the timeout and failed-delivery errors quote it. Test: `carries the crashing handler error out of a timeout`. **Acceptance item 5 — recurrence gate.** `requireRouteTable` in `modules/caddy/scripts/reconcile.ts`, used by `reconcile-routes.ts` and `setup-network.ts`. It **throws** on a non-array rather than coercing to `[]` — an empty route table is a legitimate state that renders the holding page, so coercing would converge caddy into serving nothing and drop every site block, silently. The message names the capability and the shape, which is what turns the next occurrence into an obtainable minimal repro (item 1). Both gates were watched failing under mutation, not reasoned about. **Items 1, 2 and 4 are NOT met** and should stay open: no minimal repro, no source-level fix for `{}`, and `celilo-website`'s registry poll is unverified across a full cycle. Gates: `biome` exit 0 (Checked 1930 files), `check:typecheck` 27/27, `check:modules` 0 fail, `celilo-events` 21 pass, caddy scripts 19 pass. `caddy` bumped to 3.0.1; `@celilo/cli` changeset added.