celilo/lunacycle

Fork 0

e2e-nightly has never passed: every seedDatabase() call fails on a UNIQUE constraint #32

New issue

Open

opened 2026-08-01 23:51:21 +00:00 by forgejo-admin · 1 comment

forgejo-admin commented

2026-08-01 23:51:21 +00:00

Owner

e2e-nightly.yml has run exactly once since it was created, and it failed. lunacycle currently has no working e2e coverage gating anything: release.yml deliberately does not gate on e2e (it was moved out so a Docker-runner outage couldn't block shipping), so a red nightly blocks nothing and publishes proceed regardless.

This matters more than usual right now because registry-poll CD is about to be armed for lunacycle (see #31). Once auto_upgrade is on, whatever merges to main reaches production unattended within 15 minutes, with the nightly as the only correctness signal — and it has never been green.

Symptom

Run 65 — 11 failed, 12 did not run, 6 passed, 20.4m.

Every failure is a beforeAll timeout:

"beforeAll" hook timeout of 30000ms exceeded.
  > 43 |   test.beforeAll(async () => {
    44 |     await seedDatabase('active-cycle-extra');

and the server log shows the real cause, repeated on every reseed:

[WebServer] Test reseed failed: 78 |       db.insert(schema.templateTaskUser)
[WebServer] SQLiteError: UNIQUE constraint failed:
    template_task_user.template_task_id, template_task_user.user_id

So seedDatabase() returns 500, the beforeAll hangs to its timeout, and every spec that seeds dies with it.

Scope

apps/lunacycle-server/src/test-reseed.ts — the reseed endpoint. Its delete list looks complete and correctly ordered (templateTaskUser is cleared before templateTask), so the residue theory needs proving rather than assuming.
apps/lunacycle-server/src/defaultScenario.ts:78 — the failing insert, inside createCategoryTasksAndAssignments. Note it re-reads the row it just inserted with findFirst({ where: eq(templateTask.title, task.title) }) rather than using the uuid it generated a few lines above. That lookup is by a non-unique column, so any title collision resolves to the wrong (earlier) row and re-inserts an existing (templateTaskId, userId) pair. Using the generated id directly removes the failure mode regardless of whether it's the active one here.
Concurrency is not the cause — playwright.config.ts has workers: 1 and fullyParallel: false. Ruled out.
active-cycle-extra → activeCycleWithExtraTaskScenario → activeCycleScenario, which seeds a single template task ("Meditate") for users [bob, monica]. No duplicate titles are visible in the scenario data, so the mechanism is not yet fully explained — reproduce locally before fixing.
Unrelated but adjacent: createUser awaits fetchRandomAvatar() (a network call) and then never uses the result. Three of those per seed, against an external service, inside a 30s beforeAll budget on a CI container. Worth deleting while in here.

Acceptance

Reproduce the UNIQUE-constraint reseed failure locally and name the mechanism.
seedDatabase() succeeds on repeated calls within a single server lifetime.
defaultScenario.ts no longer re-queries an inserted row by a non-unique column.
A full bun run test:playwright run is green.
e2e-nightly.yml completes green at least once on schedule (not just via workflow_dispatch).
Recurrence gate: decide and record whether a red nightly should notify anyone. It currently fails silently — nobody learned this had never passed until someone went looking at the CD wiring.

#31 — CD wiring; the PR that surfaced this while assessing whether lunacycle is safe to auto-upgrade.
.forgejo/workflows/e2e-nightly.yml — the workflow, and its header comment explaining why e2e stopped gating releases.

`e2e-nightly.yml` has run **exactly once** since it was created, and it failed. lunacycle currently has no working e2e coverage gating anything: `release.yml` deliberately does not gate on e2e (it was moved out so a Docker-runner outage couldn't block shipping), so a red nightly blocks nothing and publishes proceed regardless. This matters more than usual right now because registry-poll CD is about to be armed for lunacycle (see #31). Once `auto_upgrade` is on, whatever merges to `main` reaches production unattended within 15 minutes, with the nightly as the only correctness signal — and it has never been green. ## Symptom [Run 65](https://git.celilo.computer/celilo/lunacycle/actions/runs/65) — 11 failed, 12 did not run, 6 passed, 20.4m. Every failure is a `beforeAll` timeout: ``` "beforeAll" hook timeout of 30000ms exceeded. > 43 | test.beforeAll(async () => { 44 | await seedDatabase('active-cycle-extra'); ``` and the server log shows the real cause, repeated on every reseed: ``` [WebServer] Test reseed failed: 78 | db.insert(schema.templateTaskUser) [WebServer] SQLiteError: UNIQUE constraint failed: template_task_user.template_task_id, template_task_user.user_id ``` So `seedDatabase()` returns 500, the `beforeAll` hangs to its timeout, and every spec that seeds dies with it. ## Scope - `apps/lunacycle-server/src/test-reseed.ts` — the reseed endpoint. Its delete list *looks* complete and correctly ordered (`templateTaskUser` is cleared before `templateTask`), so the residue theory needs proving rather than assuming. - `apps/lunacycle-server/src/defaultScenario.ts:78` — the failing insert, inside `createCategoryTasksAndAssignments`. Note it re-reads the row it just inserted with `findFirst({ where: eq(templateTask.title, task.title) })` rather than using the uuid it generated a few lines above. That lookup is by a non-unique column, so any title collision resolves to the wrong (earlier) row and re-inserts an existing `(templateTaskId, userId)` pair. Using the generated id directly removes the failure mode regardless of whether it's the active one here. - Concurrency is **not** the cause — `playwright.config.ts` has `workers: 1` and `fullyParallel: false`. Ruled out. - `active-cycle-extra` → `activeCycleWithExtraTaskScenario` → `activeCycleScenario`, which seeds a single template task ("Meditate") for users `[bob, monica]`. No duplicate titles are visible in the scenario data, so the mechanism is not yet fully explained — reproduce locally before fixing. - Unrelated but adjacent: `createUser` awaits `fetchRandomAvatar()` (a network call) and then never uses the result. Three of those per seed, against an external service, inside a 30s `beforeAll` budget on a CI container. Worth deleting while in here. ## Acceptance - [ ] Reproduce the UNIQUE-constraint reseed failure locally and name the mechanism. - [ ] `seedDatabase()` succeeds on repeated calls within a single server lifetime. - [ ] `defaultScenario.ts` no longer re-queries an inserted row by a non-unique column. - [ ] A full `bun run test:playwright` run is green. - [ ] `e2e-nightly.yml` completes green at least once on schedule (not just via `workflow_dispatch`). - [ ] Recurrence gate: decide and record whether a red nightly should notify anyone. It currently fails silently — nobody learned this had never passed until someone went looking at the CD wiring. ## Related - #31 — CD wiring; the PR that surfaced this while assessing whether lunacycle is safe to auto-upgrade. - `.forgejo/workflows/e2e-nightly.yml` — the workflow, and its header comment explaining why e2e stopped gating releases.

forgejo-admin referenced this issue

2026-08-01 23:51:42 +00:00

fix(cd): document the upgrade_policy half of CD, and run the hook tests in CI #31

forgejo-admin referenced this issue from a commit

2026-08-02 04:01:52 +00:00

fix(seed): address seeded rows by their generated uuid, not by title

forgejo-admin referenced this issue

2026-08-02 04:02:16 +00:00

fix(seed): address seeded rows by their generated uuid, not by title (#32) #33

forgejo-admin referenced this issue from a commit

2026-08-02 04:03:01 +00:00

fix(seed): address seeded rows by their generated uuid, not by title

forgejo-admin referenced this issue from a commit

2026-08-02 04:06:08 +00:00

fix(seed): address seeded rows by their generated uuid, not by title

forgejo-admin commented

2026-08-02 04:06:36 +00:00

Author

Owner

Fix in #33. e2e-nightly run 71 against that branch: 29 passed, 2 skipped, 2.2m — green.

Corrections to the filing

The diagnosis was right about the site; two of the supporting claims were not.

fetchRandomAvatar() is not a network call any more. avatarUtils.ts was rewritten to generate an SVG data URI locally ("that service went away and there's no good reason for seeding to depend on the network"). So it wasn't eating the 30s budget. Its result was still computed and discarded in createUser, which is now deleted.
"Concurrency is not the cause" — workers: 1 rules out parallel tests, not overlapping requests. That's exactly the mechanism. Playwright abandons the beforeAll at 30s but the reseed it issued is still in flight server-side; the next spec's reseed clears the tables and starts seeding while the abandoned one is still writing. That's where the second same-titled Meditate comes from, which is the residue the scenario data doesn't show.
The title lookup was the right suspect, and it isn't the only one. All four self-read-backs in the seed used a non-unique column: templateTask.title, category.name, user.email, template.isActive. All four are gone; the seed mints each id up front.

Why one collision took the whole run down

A 500 makes Playwright retry the spec, which issues another reseed, which is more overlap — self-amplifying. That accounts for the symptoms that the constraint alone doesn't explain: bootstrap.spec's minimal seed (which cannot hit this bug) timing out three times, and smoke.spec.ts:16's plain request.get('http://localhost:3000/api') timing out at 30s. Both are a saturated single-threaded server, not a separate fault. Run 65 took 20.4m; run 71 takes 2.2m.

Reproduction

tests/seed/reseed.test.ts (new, runs as bun run test:seed in pr-validate.yml) seeds twice without clearing — the residue condition — and reproduces the CI error byte for byte on pre-fix code, down to the frame numbers:

SQLiteError: UNIQUE constraint failed: template_task_user.template_task_id, template_task_user.user_id
      code: "SQLITE_CONSTRAINT_PRIMARYKEY"
      at createCategoryTasksAndAssignments (.../defaultScenario.ts:83:10)
      at async activeCycleScenario (.../defaultScenario.ts:150:9)

It goes green with the fix and red again when the fix is reverted. It lives outside test:unit because the seed path needs bun:sqlite, which vitest (node) can't load.

Acceptance

Reproduce the UNIQUE-constraint reseed failure locally and name the mechanism
seedDatabase() succeeds on repeated calls within a single server lifetime
defaultScenario.ts no longer re-queries an inserted row by a non-unique column
A full bun run test:playwright run is green (locally, in the CI image, and on the runner)
e2e-nightly.yml green on schedule — run 71 was workflow_dispatch; the 07:00 UTC cron after #33 merges is the one that closes this
Recurrence gate: whether a red nightly should notify anyone — not decided here. It still fails silently, which is how this went unnoticed. Needs an owner and a channel.

Fix in #33. **[e2e-nightly run 71](https://git.celilo.computer/celilo/lunacycle/actions/runs/71) against that branch: 29 passed, 2 skipped, 2.2m — green.** ## Corrections to the filing The diagnosis was right about the site; two of the supporting claims were not. - **`fetchRandomAvatar()` is not a network call any more.** `avatarUtils.ts` was rewritten to generate an SVG data URI locally ("that service went away and there's no good reason for seeding to depend on the network"). So it wasn't eating the 30s budget. Its *result* was still computed and discarded in `createUser`, which is now deleted. - **"Concurrency is not the cause" — `workers: 1` rules out parallel *tests*, not overlapping *requests*.** That's exactly the mechanism. Playwright abandons the `beforeAll` at 30s but the reseed it issued is still in flight server-side; the next spec's reseed clears the tables and starts seeding while the abandoned one is still writing. That's where the second same-titled `Meditate` comes from, which is the residue the scenario data doesn't show. - **The `title` lookup was the right suspect, and it isn't the only one.** All four self-read-backs in the seed used a non-unique column: `templateTask.title`, `category.name`, `user.email`, `template.isActive`. All four are gone; the seed mints each id up front. ## Why one collision took the whole run down A 500 makes Playwright retry the spec, which issues another reseed, which is more overlap — self-amplifying. That accounts for the symptoms that the constraint alone doesn't explain: `bootstrap.spec`'s `minimal` seed (which cannot hit this bug) timing out three times, and `smoke.spec.ts:16`'s plain `request.get('http://localhost:3000/api')` timing out at 30s. Both are a saturated single-threaded server, not a separate fault. Run 65 took 20.4m; run 71 takes 2.2m. ## Reproduction `tests/seed/reseed.test.ts` (new, runs as `bun run test:seed` in `pr-validate.yml`) seeds twice without clearing — the residue condition — and reproduces the CI error byte for byte on pre-fix code, down to the frame numbers: ``` SQLiteError: UNIQUE constraint failed: template_task_user.template_task_id, template_task_user.user_id code: "SQLITE_CONSTRAINT_PRIMARYKEY" at createCategoryTasksAndAssignments (.../defaultScenario.ts:83:10) at async activeCycleScenario (.../defaultScenario.ts:150:9) ``` It goes green with the fix and red again when the fix is reverted. It lives outside `test:unit` because the seed path needs `bun:sqlite`, which vitest (node) can't load. ## Acceptance - [x] Reproduce the UNIQUE-constraint reseed failure locally and name the mechanism - [x] `seedDatabase()` succeeds on repeated calls within a single server lifetime - [x] `defaultScenario.ts` no longer re-queries an inserted row by a non-unique column - [x] A full `bun run test:playwright` run is green (locally, in the CI image, and on the runner) - [ ] `e2e-nightly.yml` green **on schedule** — run 71 was `workflow_dispatch`; the 07:00 UTC cron after #33 merges is the one that closes this - [ ] Recurrence gate: whether a red nightly should notify anyone — **not decided here.** It still fails silently, which is how this went unnoticed. Needs an owner and a channel.

forgejo-admin referenced this issue from a commit

2026-08-02 05:15:26 +00:00

Merge pull request 'fix(seed): address seeded rows by their generated uuid, not by title (#32)' (#33) from fix-e2e-seed-unique into main