ci: stop gating publish on e2e; move e2e to a nightly job #26

Merged

forgejo-admin merged 1 commit from ci-decouple-e2e-from-publish into main

2026-07-31 20:56:00 +00:00

forgejo-admin commented

2026-07-31 20:52:21 +00:00

Owner

The full Playwright suite gated publish, so a Docker-runner outage blocked shipping entirely rather than just costing test coverage. release.yml's own header admitted it: "until it's up, publish (which needs e2e) stays blocked."

What changes

release.yml is now version → publish. publish depends only on version, so a merge to main with no pending changesets builds a fresh .netapp and uploads it to celilo-registry. That is where the pipeline ends.

The pull_request trigger is removed as well — it existed only to give e2e pre-merge feedback, and without e2e a PR run would do nothing but detect changesets. pr-validate.yml is untouched, so every PR still gets lint, typecheck and unit tests.

e2e moves to e2e-nightly.yml: 07:00 UTC daily, plus workflow_dispatch so it can still be run against a branch on demand when a change warrants it. Same container, runner and steps as the job it replaces.

Net effect on triggers:

	before	after
PR	pr-validate + release(version, e2e)	pr-validate
push to main	version, e2e, publish	version, publish
nightly	—	e2e
on demand	—	e2e (dispatch)

Two stale claims corrected in the header

It said celilo-mgr's poll-CD upgrades the fleet after publish. The registry-poll subscription does exist in celilo-mgmt (timer.tick.15m → celilo module upgrade), but it does not currently execute on celilo-mgr — timer.tick.15m fires (27 in the last 300 events) and the dispatcher is healthy, yet there is not a single registry-poll or module upgrade entry in the event log. That is being investigated separately in the celilo repo.
The opt-in that poll keys off, auto_upgrade, is per-install operator config, not a manifest field. I tried setting it in manifest.yml and backed it out: the top-level manifest schema is .strict() (apps/celilo/src/manifest/schema.ts:834) and declares no such key, so it would be rejected at validation. The working path is sudo celilo module config set lunacycle auto_upgrade true, which resolveAutoUpgrade checks first via a plain config-DB lookup. It defaults to false.

The header now states where the pipeline ends and what has to be true separately for a rollout to follow, rather than implying deploys are automatic.

Merge order

Worth merging this before #21 (chore: version lunacycle). #21 is what triggers the publish of the smoke-check fixes from #20; if it merges first, that publish still has to clear the e2e gate on the Docker runner.

Not verified

I could not exercise these workflows — no Forgejo Actions runner available from here. All three files parse and the job graph is confirmed (publish → needs: ["version"], gated to push), but the first real run is the proof. The cron syntax and workflow_dispatch on a schedule-triggered workflow are the parts I would watch on the first nightly.

I also dropped a Playwright-report artifact upload I had drafted: actions/upload-artifact needs Forgejo artifact storage configured, which I could not confirm, and a failing upload under if: always() would turn a green suite red. Easy to add once someone can verify it works.

The full Playwright suite gated `publish`, so a Docker-runner outage blocked shipping entirely rather than just costing test coverage. `release.yml`'s own header admitted it: *"until it's up, `publish` (which needs e2e) stays blocked."* ## What changes **`release.yml` is now `version` → `publish`.** `publish` depends only on `version`, so a merge to main with no pending changesets builds a fresh `.netapp` and uploads it to celilo-registry. That is where the pipeline ends. The `pull_request` trigger is removed as well — it existed only to give e2e pre-merge feedback, and without e2e a PR run would do nothing but detect changesets. **`pr-validate.yml` is untouched**, so every PR still gets lint, typecheck and unit tests. **e2e moves to `e2e-nightly.yml`:** 07:00 UTC daily, plus `workflow_dispatch` so it can still be run against a branch on demand when a change warrants it. Same container, runner and steps as the job it replaces. Net effect on triggers: | | before | after | |---|---|---| | PR | pr-validate + release(version, e2e) | pr-validate | | push to main | version, e2e, publish | version, publish | | nightly | — | e2e | | on demand | — | e2e (dispatch) | ## Two stale claims corrected in the header 1. It said celilo-mgr's poll-CD upgrades the fleet after publish. The `registry-poll` subscription does exist in celilo-mgmt (`timer.tick.15m` → `celilo module upgrade`), but it **does not currently execute** on celilo-mgr — `timer.tick.15m` fires (27 in the last 300 events) and the dispatcher is healthy, yet there is not a single `registry-poll` or `module upgrade` entry in the event log. That is being investigated separately in the celilo repo. 2. The opt-in that poll keys off, `auto_upgrade`, is **per-install operator config, not a manifest field**. I tried setting it in `manifest.yml` and backed it out: the top-level manifest schema is `.strict()` (`apps/celilo/src/manifest/schema.ts:834`) and declares no such key, so it would be rejected at validation. The working path is `sudo celilo module config set lunacycle auto_upgrade true`, which `resolveAutoUpgrade` checks first via a plain config-DB lookup. It defaults to `false`. The header now states where the pipeline ends and what has to be true separately for a rollout to follow, rather than implying deploys are automatic. ## Merge order Worth merging this **before** [#21](https://git.celilo.computer/celilo/lunacycle/pulls/21) (`chore: version lunacycle`). #21 is what triggers the publish of the smoke-check fixes from #20; if it merges first, that publish still has to clear the e2e gate on the Docker runner. ## Not verified I could not exercise these workflows — no Forgejo Actions runner available from here. All three files parse and the job graph is confirmed (`publish → needs: ["version"]`, gated to `push`), but the first real run is the proof. The cron syntax and `workflow_dispatch` on a schedule-triggered workflow are the parts I would watch on the first nightly. I also dropped a Playwright-report artifact upload I had drafted: `actions/upload-artifact` needs Forgejo artifact storage configured, which I could not confirm, and a failing upload under `if: always()` would turn a green suite red. Easy to add once someone can verify it works.

forgejo-admin added 1 commit

2026-07-31 20:52:21 +00:00

ci: stop gating publish on e2e; move e2e to a nightly job

pr-validate / validate (pull_request) Successful in 14s

Details

6e64ec72b9

The full Playwright suite gated `publish`, so a Docker-runner outage blocked
shipping entirely rather than just costing test coverage — and the release
workflow's own header acknowledged as much ("until it's up, publish (which
needs e2e) stays blocked").

release.yml is now version -> publish. `publish` depends only on `version`, so
a merge to main with no pending changesets builds a fresh .netapp and uploads
it to celilo-registry, full stop. The pull_request trigger goes too: it existed
only to give e2e pre-merge feedback, and without e2e a PR run would do nothing
but detect changesets. pr-validate.yml still runs lint, typecheck and unit
tests on every PR, unchanged.

e2e moves to e2e-nightly.yml: 07:00 UTC daily plus workflow_dispatch, so it can
still be run against a branch on demand when a change warrants it. Same
container, runner and steps as the job it replaces.

Also corrects two stale claims in the release.yml header. It said celilo-mgr's
poll-CD upgrades the fleet after publish; the registry-poll subscription exists
in celilo-mgmt but does not currently execute on celilo-mgr, and the opt-in it
keys off (`auto_upgrade`) is per-install operator config rather than a manifest
field — the manifest schema is strict() and does not declare it. The header now
says where the pipeline ends (a .netapp in the registry) and what has to be true
separately for a rollout to follow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

forgejo-admin merged commit 985510a271 into main

2026-07-31 20:56:00 +00:00

forgejo-admin referenced this pull request from a commit

2026-07-31 20:56:02 +00:00

Merge pull request 'ci: stop gating publish on e2e; move e2e to a nightly job' (#26) from ci-decouple-e2e-from-publish into main

forgejo-admin referenced this pull request

2026-07-31 20:58:46 +00:00

feat(manifest): declare auto_upgrade so it can be set #27

forgejo-admin commented

2026-07-31 21:44:03 +00:00

Author

Owner

Correction to this PR's description, for anyone reading it later.

I wrote that the registry-poll subscription "does not currently execute" on celilo-mgr, on the evidence that timer.tick.15m fires while no registry-poll or module upgrade entry appears in the event log. That conclusion was wrong.

The handler has been executing every 15 minutes and failing identically every time, since it was wired. Root cause (found in the celilo repo, branch fix-poll-cd-not-running):

The dispatcher invokes a subprocess handler as <handler> <event_id>. celilo-mgmt's subscription is handler: celilo module upgrade, so what actually runs each tick is:

celilo module upgrade 5517

module upgrade's module name is an optional positional and "no name = run the poll" — so the appended event id lands in that slot, and it looks up a module literally named 5517. Error: Module not found: 5517, exit 1, every tick. celilo events list-subscribers does contain the subscription (id 14); registration was never the problem.

The reason it was invisible, and the reason my evidence looked conclusive: the delivery error was a bare handler exited with code 1 with no cause. The dispatcher captures a stderr tail, but the celilo CLI renders errors to stdout via clack, so the tail was empty every time. A handler failing identically every 15m is indistinguishable from one that never fired.

The observable outcome this PR was written around is unchanged — nothing auto-deploys, so publish should not have been gated on a runner that could block shipping. But "the subscription never runs" was the wrong reason, and the fix is in celilo (an explicit --poll flag immune to the trailing event id, plus a dispatcher fallback to the stdout tail), not here.

Correction to this PR's description, for anyone reading it later. I wrote that the `registry-poll` subscription **"does not currently execute"** on celilo-mgr, on the evidence that `timer.tick.15m` fires while no `registry-poll` or `module upgrade` entry appears in the event log. That conclusion was wrong. The handler **has been executing every 15 minutes and failing identically every time**, since it was wired. Root cause (found in the celilo repo, branch `fix-poll-cd-not-running`): The dispatcher invokes a subprocess handler as `<handler> <event_id>`. celilo-mgmt's subscription is `handler: celilo module upgrade`, so what actually runs each tick is: ``` celilo module upgrade 5517 ``` `module upgrade`'s module name is an optional positional and "no name = run the poll" — so the appended event id lands in that slot, and it looks up a module literally named `5517`. `Error: Module not found: 5517`, exit 1, every tick. `celilo events list-subscribers` does contain the subscription (id 14); registration was never the problem. The reason it was invisible, and the reason my evidence looked conclusive: the delivery error was a bare `handler exited with code 1` with no cause. The dispatcher captures a **stderr** tail, but the celilo CLI renders errors to **stdout** via clack, so the tail was empty every time. A handler failing identically every 15m is indistinguishable from one that never fired. The observable outcome this PR was written around is unchanged — nothing auto-deploys, so publish should not have been gated on a runner that could block shipping. But "the subscription never runs" was the wrong reason, and the fix is in celilo (an explicit `--poll` flag immune to the trailing event id, plus a dispatcher fallback to the stdout tail), not here.