celilo/celilo

Fork 0

npm-consumer-smoke's bake step times out waiting for install.sh over HTTPS on celilo-builder, so no full smoke run has passed today and the release's version PR cannot merge #1365

New issue

Open

opened 2026-09-10 20:26:39 +00:00 by forgejo-admin · 4 comments

forgejo-admin commented

2026-09-10 20:26:39 +00:00

Owner

Every npm-consumer-smoke run on celilo-builder today that did real work has failed. The two that got past the orphan-stack refusal both died at the same place, after a clean shared-infra start. The log reads Timed out (30s) waiting for celilo.computer to serve install.sh over HTTPS, and goes on to name a stalled Pebble CA fetch or a down website-sim as the likely cause. Every green smoke run today took the 2 to 4 second path that skips work for docs-only changes, so none of them measured the bake. This check gates the release's Version Packages PR, so the release is blocked on it.

Evidence

Run 6666 (polecat/ce-ojdn at 3385b83d, 19:00Z). Shared infra cleaned and started, then the bake failed at 19:01:28Z after 80s.
Run 6682 (main at 82e88e58, dispatched at 20:19Z). Shared infra cleaned and started, then the bake failed at 20:21:12Z after 81s, with the same message.
/api/v1/repos/celilo/celilo/actions/tasks, every npm-consumer-smoke task since 15:33Z. Each success lasted 2 to 4 seconds. Each run that did real work lasted 23 to 81 seconds and failed. The 23 to 34 second ones are the orphan-age refusal (celilo#1320, #1314). The 80 second ones are this bake timeout.

Scope

The bake step in packages/e2e: the build-infra install.sh bake, the website-sim, and Pebble. The log itself points at cele2e run install-sh for a focused reproduction. That needs a rig, so go through the mayor's e2e lock on the Mac, or reproduce on the builder.

Acceptance

The cause is measured and recorded here: the Pebble CA fetch, the website-sim not serving install.sh, or DNS or routing inside the rig.
A full npm-consumer-smoke run passes on main.

celilo#1320, #1314, #1302 and #1352: orphaned stacks and misleading refusals on the builder. celilo#1318 (closed): an earlier bake breakage.
It blocks the release in bead ce-42q4.1 (crew-builder). Bead to follow in a comment.

Every `npm-consumer-smoke` run on celilo-builder today that did real work has failed. The two that got past the orphan-stack refusal both died at the same place, after a clean shared-infra start. The log reads `Timed out (30s) waiting for celilo.computer to serve install.sh over HTTPS`, and goes on to name a stalled Pebble CA fetch or a down website-sim as the likely cause. Every green smoke run today took the 2 to 4 second path that skips work for docs-only changes, so none of them measured the bake. This check gates the release's Version Packages PR, so the release is blocked on it. ## Evidence - Run 6666 (`polecat/ce-ojdn` at `3385b83d`, 19:00Z). Shared infra cleaned and started, then the bake failed at 19:01:28Z after 80s. - Run 6682 (`main` at `82e88e58`, dispatched at 20:19Z). Shared infra cleaned and started, then the bake failed at 20:21:12Z after 81s, with the same message. - `/api/v1/repos/celilo/celilo/actions/tasks`, every `npm-consumer-smoke` task since 15:33Z. Each success lasted 2 to 4 seconds. Each run that did real work lasted 23 to 81 seconds and failed. The 23 to 34 second ones are the orphan-age refusal (celilo#1320, #1314). The 80 second ones are this bake timeout. ## Scope - The bake step in `packages/e2e`: the build-infra `install.sh` bake, the website-sim, and Pebble. The log itself points at `cele2e run install-sh` for a focused reproduction. That needs a rig, so go through the mayor's e2e lock on the Mac, or reproduce on the builder. ## Acceptance - [ ] The cause is measured and recorded here: the Pebble CA fetch, the website-sim not serving `install.sh`, or DNS or routing inside the rig. - [ ] A full `npm-consumer-smoke` run passes on `main`. ## Related - celilo#1320, #1314, #1302 and #1352: orphaned stacks and misleading refusals on the builder. celilo#1318 (closed): an earlier bake breakage. - It blocks the release in bead ce-42q4.1 (crew-builder). Bead to follow in a comment.

forgejo-admin added the

category:bug

severity:high

labels

2026-09-10 20:27:07 +00:00

forgejo-admin commented

2026-09-10 20:28:24 +00:00

Author

Owner

Bead: ce-94oi (celilo store, P1). It is queued as NEXT BATCH (hold:mayor) because polecats are running, and the mayor lifts the batch when the pool empties. It carries the same content as this issue.

Bead: `ce-94oi` (celilo store, P1). It is queued as NEXT BATCH (`hold:mayor`) because polecats are running, and the mayor lifts the batch when the pool empties. It carries the same content as this issue.

forgejo-admin commented

2026-09-10 21:45:46 +00:00

Author

Owner

Progress from polecat work on ce-94oi (analysis so far; reproduction continuing).

Two failure families on the builder, one underlying cycle.

The genuine failure: the bake's 30s HTTPS probe times out (runs 6636, 6666, 6682, 6683, 6698). DNS is NOT the cause — run 6698's new self-diagnosing probe (branch polecat/ce-94oi, commit 24546d16) shows both resolvers answer celilo.computer → 100.64.0.58 instantly. The failure is TLS: curl: (60) SSL certificate problem: unable to get local issuer certificate. The website-sim serves a certificate; management cannot verify it. A full build-infra run on the Mac rig with the same code PASSES (pebble CA ready ✔ 5s, Baked in 47s), so this is builder-specific or a startup-timing race the Mac does not hit. Prime suspect: management's Pebble runtime-root fetch (management-routes.sh foreground loop, worst case 120s) — if Pebble's :15000 is slow to answer on the builder, the whole 30s probe window runs against the static repo CA only, which never matches the fresh runtime root.
The refusal family: after a failed bake, the job dies at the build-infra step and the if: always() teardown steps never run (successful run 6523 logged "shared stack torn down"; every failing run — 6645, 6682, 6693, 6698 — ends at "Cleaning up container for job" with no teardown output). The 13-container shared stack then sits live on the builder for 60 minutes and every subsequent full run refuses at startup cleanup (6693: "youngest container is only 31m old"). Cycle: bake timeout → job dies before teardown → 60m of refusals → stack ages past the orphan threshold → next full run sweeps it → bake runs → times out again.

Next step: a builder bake with the extended diagnostics (trusted CA file listing, a direct Pebble :15000 probe from the management container, openssl s_client chain dump), dispatched once the leaked stack ages past the orphan gate (~22:25Z).

Progress from polecat work on ce-94oi (analysis so far; reproduction continuing). **Two failure families on the builder, one underlying cycle.** 1. The genuine failure: the bake's 30s HTTPS probe times out (runs 6636, 6666, 6682, 6683, 6698). DNS is NOT the cause — run 6698's new self-diagnosing probe (branch polecat/ce-94oi, commit 24546d16) shows both resolvers answer `celilo.computer → 100.64.0.58` instantly. The failure is TLS: `curl: (60) SSL certificate problem: unable to get local issuer certificate`. The website-sim serves a certificate; management cannot verify it. A full build-infra run on the Mac rig with the same code PASSES (pebble CA ready ✔ 5s, Baked in 47s), so this is builder-specific or a startup-timing race the Mac does not hit. Prime suspect: management's Pebble runtime-root fetch (management-routes.sh foreground loop, worst case 120s) — if Pebble's :15000 is slow to answer on the builder, the whole 30s probe window runs against the static repo CA only, which never matches the fresh runtime root. 2. The refusal family: after a failed bake, the job dies at the build-infra step and the `if: always()` teardown steps never run (successful run 6523 logged "shared stack torn down"; every failing run — 6645, 6682, 6693, 6698 — ends at "Cleaning up container for job" with no teardown output). The 13-container shared stack then sits live on the builder for 60 minutes and every subsequent full run refuses at startup cleanup (6693: "youngest container is only 31m old"). Cycle: bake timeout → job dies before teardown → 60m of refusals → stack ages past the orphan threshold → next full run sweeps it → bake runs → times out again. Next step: a builder bake with the extended diagnostics (trusted CA file listing, a direct Pebble :15000 probe from the management container, openssl s_client chain dump), dispatched once the leaked stack ages past the orphan gate (~22:25Z).

forgejo-admin referenced this issue

2026-09-10 21:47:14 +00:00

npm-consumer-smoke can only pass once an hour: its job-end teardown is blocked by any bun test on the builder, so the next run inside 60m refuses on the orphan age gate #1320

forgejo-admin referenced this issue

2026-09-20 23:19:48 +00:00

e2e base images pull floating :latest tags, and minio's two are now delisted from Docker Hub, so every cold-cache bake fails #1389

forgejo-admin referenced this issue

2026-09-20 23:20:03 +00:00

cele2e doctor green-lights a hollow management image: it checks the container starts, not that celilo is in it #1390

forgejo-admin commented

2026-09-20 23:21:41 +00:00

Author

Owner

Reproduced on a second machine, 2026-09-20, outside CI

The same bake timeout hit an agent bringing up e2e for the bna-yard-sale-2 module on peba's Mac, not on celilo-builder:

Timed out (30s) waiting for celilo.computer to serve install.sh over HTTPS

So this is not specific to the builder or to npm-consumer-smoke. Suspected locally as a stalled Pebble CA fetch or a down website-sim, unverified.

What it costs downstream, which was not visible from CI. The timeout does not abort the bake. It leaves the management image hollow: built, startable, and carrying no celilo binary.

docker run --rm --entrypoint sh celilo-e2e/management:latest -c 'command -v celilo'
# no output

cele2e doctor then reported the environment runnable, and the suite died much later at stage 1 with celilo --version exited 127: command not found (0 passed, 1 failed, 9 stages skipped). Whoever fixes this should know the failure currently resurfaces in a disguise several minutes downstream.

Filed alongside, both with beads:

#1390 — doctor green-lights the hollow image this failure produces (bead ce-xtde).
#1389 — cold-cache bakes also fail on delisted minio base images (bead ce-p52l).

A cold-cache user hits all three in sequence, and only the last one produces an error message that mentions anything they did.

## Reproduced on a second machine, 2026-09-20, outside CI The same bake timeout hit an agent bringing up e2e for the `bna-yard-sale-2` module on peba's Mac, not on celilo-builder: ``` Timed out (30s) waiting for celilo.computer to serve install.sh over HTTPS ``` So this is not specific to the builder or to `npm-consumer-smoke`. Suspected locally as a stalled Pebble CA fetch or a down website-sim, unverified. **What it costs downstream, which was not visible from CI.** The timeout does not abort the bake. It leaves the management image **hollow**: built, startable, and carrying no celilo binary. ``` docker run --rm --entrypoint sh celilo-e2e/management:latest -c 'command -v celilo' # no output ``` `cele2e doctor` then reported the environment runnable, and the suite died much later at stage 1 with `celilo --version exited 127: command not found` (0 passed, 1 failed, 9 stages skipped). Whoever fixes this should know the failure currently resurfaces in a disguise several minutes downstream. Filed alongside, both with beads: - #1390 — doctor green-lights the hollow image this failure produces (bead `ce-xtde`). - #1389 — cold-cache bakes also fail on delisted minio base images (bead `ce-p52l`). A cold-cache user hits all three in sequence, and only the last one produces an error message that mentions anything they did.

forgejo-admin referenced this issue

2026-09-21 01:20:37 +00:00

Guard the forgejo-runner binary download with creates:, so a transient egress blip cannot fail a deploy that needs no download #1394

forgejo-admin commented

2026-09-21 02:56:05 +00:00

Author

Owner

Hit again 2026-09-21, and it did exactly what this issue's title predicts: it red-flagged the release's version PR

Run 6954, on the chore: version packages PR #1396 for the @celilo/cli 3.1.0 → 4.0.0 release:

error: Timed out (30s) waiting for celilo.computer to serve install.sh over HTTPS
       — Pebble CA fetch likely stalled or website-sim is down.
✗  59s
Bake step failed after 59s
Job 'npm-consumer-smoke' failed

Why this one is worth recording: it is demonstrably intermittent, and I have the contrast case

npm-consumer-smoke succeeded on afc6ef0b roughly an hour earlier the same evening, on the same builder, with the same website-sim. So the consumer path itself is fine; this is the flake.

Separately the same day, a cele2e build-infra hit this identical install.sh HTTPS timeout once and then succeeded on retry with nothing changed. Three data points now agree it is transient.

The cost, concretely

The rollup status for PR #1396 went failure while ci / validate and all three pr-guards were still pending. So the only red check was this one, and it makes the PR read as broken when nothing about the version bump is. A version PR bumps package.json, CHANGELOGs, consumer pins and bun.lock — none of which this check is testing.

Worse for an automated or unattended release: the natural read of "rollup = failure" is stop, and the correct read is this specific check is the known flake, look at the others. That distinction is not visible without opening the log, which is the same shape as the failure-means-cancelled trap already documented in CLAUDE.md for commit statuses on main.

Suggestion, not a fix

Given it gates releases and is transient, it may be worth deciding deliberately whether npm-consumer-smoke should be required on the version PR at all, or whether the bake step should retry the install.sh fetch before failing. A 30s timeout against a simulated CA fetch on a loaded builder is a narrow window. Either change would have let this release proceed without a human judgement call.

Recording only; no acceptance criterion here is met.

#1376 — pin the e2e base images / stop anonymous Docker Hub pulls; same job, adjacent fragility.

## Hit again 2026-09-21, and it did exactly what this issue's title predicts: it red-flagged the release's version PR Run [6954](https://git.celilo.computer/celilo/celilo/actions/runs/6954), on the `chore: version packages` PR #1396 for the `@celilo/cli` 3.1.0 → 4.0.0 release: ``` error: Timed out (30s) waiting for celilo.computer to serve install.sh over HTTPS — Pebble CA fetch likely stalled or website-sim is down. ✗ 59s Bake step failed after 59s Job 'npm-consumer-smoke' failed ``` ### Why this one is worth recording: it is demonstrably intermittent, and I have the contrast case `npm-consumer-smoke` **succeeded** on `afc6ef0b` roughly an hour earlier the same evening, on the same builder, with the same website-sim. So the consumer path itself is fine; this is the flake. Separately the same day, a `cele2e build-infra` hit this identical `install.sh` HTTPS timeout once and then **succeeded on retry with nothing changed**. Three data points now agree it is transient. ### The cost, concretely The rollup status for PR #1396 went `failure` while `ci / validate` and all three `pr-guards` were still `pending`. So the only red check was this one, and it makes the PR read as broken when nothing about the version bump is. A version PR bumps `package.json`, CHANGELOGs, consumer pins and `bun.lock` — none of which this check is testing. Worse for an automated or unattended release: the natural read of "rollup = failure" is *stop*, and the correct read is *this specific check is the known flake, look at the others*. That distinction is not visible without opening the log, which is the same shape as the `failure`-means-cancelled trap already documented in CLAUDE.md for commit statuses on main. ### Suggestion, not a fix Given it gates releases and is transient, it may be worth deciding deliberately whether `npm-consumer-smoke` should be **required** on the version PR at all, or whether the bake step should retry the `install.sh` fetch before failing. A 30s timeout against a simulated CA fetch on a loaded builder is a narrow window. Either change would have let this release proceed without a human judgement call. Recording only; no acceptance criterion here is met. ### Related - #1376 — pin the e2e base images / stop anonymous Docker Hub pulls; same job, adjacent fragility.