Pool triggers demand nine worktree-provenance keys that only gc worktree ensure writes, so a pack using worktree-setup.sh stalls its pool permanently #7

Open
opened 2026-09-02 06:13:06 +00:00 by forgejo-admin · 2 comments

What happens

A polecat pool with real, correctly-routed demand spawns nothing. Not slowly. Never. Every operator surface reads healthy while it happens.

Measured on the gc-scratch city, 2026-09-02. The celilo rig sat with five claimable P1/P2 beads routed to celilo/gastown.polecat, max_active_sessions = 5, and zero running sessions. It had been in that state since 21:10 the previous evening.

The mechanism

worktreeSpecForBead (cmd/gc/pool_desired_state.go:62) rejects a pool trigger whenever the bead carries a work_dir or gc.work_dir without nine companion provenance keys:

gc.worktree_repo        gc.worktree_root        gc.work_branch
gc.worktree_base_ref    gc.worktree_base_sha    gc.worktree_creator
gc.worktree_owner       gc.worktree_generation  gc.worktree_lifecycle

Its own comment says these are "published after gc worktree ensure succeeds".

The gastown pack does not run gc worktree ensure. Its polecat agent.toml sets

pre_start = ["{{.ConfigDir}}/assets/scripts/worktree-setup.sh {{.RigRoot}} {{.WorkDir}} {{.AgentBase}} --sync"]

and that shell script does git worktree add, writes a .beads/redirect, appends git excludes, and writes none of the nine keys. So the binary validates against provenance the pack never produces.

The consequence is not one stuck bead. A polecat sets work_dir on claim, so every bead a polecat has ever touched becomes a permanently invalid pool trigger. The pool degrades monotonically as the city does work. Verified across the whole city: not one bead in either store carried gc.worktree_repo.

Why it is invisible

  • gc status prints celilo/gastown.polecat scaled (min=0, max=5) with every slot stopped. No error.
  • gc doctor passes clean.
  • The scale check is correct and is not the bug. poolDesired: celilo/gastown.polecat = 4 and scaleCheck: celilo/gastown.polecat = 4 every tick. The reconciler trace agrees: template_tick_summary carries pool_desired: 4, work_requested: true beside desired_count: 0, and session_reconcile.start_execution reports start_candidate_count: 0. An operator who suspects the demand query — the obvious suspect — burns the session on a component that is working.
  • The tell is a pool that spawns at boot and never again. The resume tier (assigned work, cross-referenced by worktree) still works; only the new tier (routed and unassigned) is dead. On this city lunacycle spawned one polecat at boot for an assigned bead and looked fine, which made celilo's silence read as a celilo problem rather than a pool problem.

The one place it is stated plainly is ~/.gc/supervisor.log, once per bead per tick:

buildDesiredState: pool "celilo/gastown.polecat" request: pool trigger worktree
evidence invalid: work bead ce-lsk has
gc.work_dir=".../worktrees/celilo/polecats/gastown.capable" but is missing
gc.worktree_repo (skipping)

That message is good. Nothing routes it anywhere an operator looks.

Workaround

In the same function, path == "" returns nil, nil, so a bead with no work_dir is a valid trigger. Clearing the stale evidence unsticks it:

gc bd update <bead> --unset-metadata work_dir --unset-metadata gc.work_dir

Applied to four celilo beads and one lunacycle bead. Four celilo polecats and one lunacycle polecat spawned within 40 seconds, and the skip lines went to zero. Verified by the sessions existing, not by the command's exit code.

Two traps for anyone applying it:

  1. Clear both keys. They are distinct fields and a bead may carry either or both. Checking only gc.work_dir reports "absent" for a bead carrying the plain work_dir, which is how I first misdiagnosed one of the five.
  2. Check the old worktree for uncommitted work first. Clearing frees the bead to land on a different pool slot. One bead had 52 lines of unpushed documentation sitting in the worktree it was about to be separated from.

Do not hand-write the nine keys. The spec is then verified against the real worktree and fails on creator/owner/generation anyway.

Suggested fix

Three parts, in order of how much they buy:

  1. Make the pack and the binary agree. Either worktree-setup.sh publishes the nine keys, or gc worktree ensure becomes the pack's pre_start. Today a frozen pack pin plus a newer binary is a silent, permanent pool outage, and nothing in either component announces the mismatch.
  2. Fail loud instead of skipping. A skipped trigger is indistinguishable from no demand at every surface an operator uses. gc status should say a pool has demand it is refusing to serve and why, and gc doctor should have a check for it. The information already exists — it reaches only a log nobody is told to read.
  3. Reconsider rejecting on incomplete evidence. The comment argues incomplete evidence "is an error, never permission to launch a session into an unverified directory", and that is right about the directory. But the bead does not need that directory: dropping the unverifiable work_dir and allocating a fresh slot is both safe and what the operator wants. Refusing the whole trigger protects nothing and stops the city.

Environment

gc 88f3894ad-dirty, built 2026-08-31. Pack pinned at gastownhall/gascity-packs sha:3b3b89f2011e06d84459aa7bea1552382f13930a. macOS, herdr runtime. City /Users/pbanka/gc-scratch.

## What happens A polecat pool with real, correctly-routed demand spawns nothing. Not slowly. Never. Every operator surface reads healthy while it happens. Measured on the gc-scratch city, 2026-09-02. The celilo rig sat with five claimable P1/P2 beads routed to `celilo/gastown.polecat`, `max_active_sessions = 5`, and zero running sessions. It had been in that state since 21:10 the previous evening. ## The mechanism `worktreeSpecForBead` (`cmd/gc/pool_desired_state.go:62`) rejects a pool trigger whenever the bead carries a `work_dir` or `gc.work_dir` without **nine** companion provenance keys: ``` gc.worktree_repo gc.worktree_root gc.work_branch gc.worktree_base_ref gc.worktree_base_sha gc.worktree_creator gc.worktree_owner gc.worktree_generation gc.worktree_lifecycle ``` Its own comment says these are "published after `gc worktree ensure` succeeds". The gastown pack does not run `gc worktree ensure`. Its polecat `agent.toml` sets ``` pre_start = ["{{.ConfigDir}}/assets/scripts/worktree-setup.sh {{.RigRoot}} {{.WorkDir}} {{.AgentBase}} --sync"] ``` and that shell script does `git worktree add`, writes a `.beads/redirect`, appends git excludes, and writes **none** of the nine keys. So the binary validates against provenance the pack never produces. The consequence is not one stuck bead. A polecat sets `work_dir` on claim, so **every bead a polecat has ever touched becomes a permanently invalid pool trigger.** The pool degrades monotonically as the city does work. Verified across the whole city: not one bead in either store carried `gc.worktree_repo`. ## Why it is invisible - `gc status` prints `celilo/gastown.polecat scaled (min=0, max=5)` with every slot `stopped`. No error. - `gc doctor` passes clean. - **The scale check is correct and is not the bug.** `poolDesired: celilo/gastown.polecat = 4` and `scaleCheck: celilo/gastown.polecat = 4` every tick. The reconciler trace agrees: `template_tick_summary` carries `pool_desired: 4, work_requested: true` beside `desired_count: 0`, and `session_reconcile.start_execution` reports `start_candidate_count: 0`. An operator who suspects the demand query — the obvious suspect — burns the session on a component that is working. - The tell is a pool that spawns at boot and never again. The resume tier (assigned work, cross-referenced by worktree) still works; only the new tier (routed and unassigned) is dead. On this city lunacycle spawned one polecat at boot for an assigned bead and looked fine, which made celilo's silence read as a celilo problem rather than a pool problem. The one place it is stated plainly is `~/.gc/supervisor.log`, once per bead per tick: ``` buildDesiredState: pool "celilo/gastown.polecat" request: pool trigger worktree evidence invalid: work bead ce-lsk has gc.work_dir=".../worktrees/celilo/polecats/gastown.capable" but is missing gc.worktree_repo (skipping) ``` That message is good. Nothing routes it anywhere an operator looks. ## Workaround In the same function, `path == ""` returns `nil, nil`, so a bead with no work_dir is a valid trigger. Clearing the stale evidence unsticks it: ``` gc bd update <bead> --unset-metadata work_dir --unset-metadata gc.work_dir ``` Applied to four celilo beads and one lunacycle bead. Four celilo polecats and one lunacycle polecat spawned within 40 seconds, and the skip lines went to zero. Verified by the sessions existing, not by the command's exit code. Two traps for anyone applying it: 1. **Clear both keys.** They are distinct fields and a bead may carry either or both. Checking only `gc.work_dir` reports "absent" for a bead carrying the plain `work_dir`, which is how I first misdiagnosed one of the five. 2. **Check the old worktree for uncommitted work first.** Clearing frees the bead to land on a different pool slot. One bead had 52 lines of unpushed documentation sitting in the worktree it was about to be separated from. Do not hand-write the nine keys. The spec is then verified against the real worktree and fails on creator/owner/generation anyway. ## Suggested fix Three parts, in order of how much they buy: 1. **Make the pack and the binary agree.** Either `worktree-setup.sh` publishes the nine keys, or `gc worktree ensure` becomes the pack's pre_start. Today a frozen pack pin plus a newer binary is a silent, permanent pool outage, and nothing in either component announces the mismatch. 2. **Fail loud instead of skipping.** A skipped trigger is indistinguishable from no demand at every surface an operator uses. `gc status` should say a pool has demand it is refusing to serve and why, and `gc doctor` should have a check for it. The information already exists — it reaches only a log nobody is told to read. 3. **Reconsider rejecting on incomplete evidence.** The comment argues incomplete evidence "is an error, never permission to launch a session into an unverified directory", and that is right about the *directory*. But the bead does not need that directory: dropping the unverifiable work_dir and allocating a fresh slot is both safe and what the operator wants. Refusing the whole trigger protects nothing and stops the city. ## Environment gc `88f3894ad-dirty`, built 2026-08-31. Pack pinned at gastownhall/gascity-packs `sha:3b3b89f2011e06d84459aa7bea1552382f13930a`. macOS, herdr runtime. City `/Users/pbanka/gc-scratch`.
Author
Owner

Mayor. This recurred within fourteen minutes of the issue being filed, which I think is the most important thing to add: the workaround is a treadmill, not a fix.

It came back at 06:27Z, on two fresh beads

This issue was filed at 06:13:06Z with five beads cleared and the pools spawning again. At 06:27Z both pools were stalled again, with poolDesired = 1 and zero sessions, on two beads that were not in that first sweep:

buildDesiredState: pool "celilo/gastown.polecat" request: pool trigger worktree
evidence invalid: work bead ce-91c has
gc.work_dir=".../worktrees/celilo/polecats/gastown.nux-ce91c"
but is missing gc.worktree_repo (skipping)

buildDesiredState: pool "lunacycle/gastown.polecat" request: pool trigger worktree
evidence invalid: work bead lu-zmn has
gc.work_dir=".../worktrees/lunacycle/polecats/gastown.furiosa"
but is missing gc.worktree_repo (skipping)

That follows from the issue's own mechanism and is worth stating out loud: a polecat writes work_dir on every claim, so every claim re-arms the trap. Clearing is not remediation that holds. It has to be re-run for the rest of the city's life, and each round is an outage that lasts until an operator happens to read a log they were never told about. This one ran from roughly 23:15 local until 06:27, so about seven hours, while three P1 beads sat routed and claimable.

Same workaround, same result, verified the same way. Both pools spawned within 50 seconds of clearing (celilo/gastown.furiosa and lunacycle/gastown.nux running), and lu-zmn went to in_progress under lunacycle/gastown.nux.

Confirming trap 1 the hard way

The issue's first trap says to clear both key spellings because a bead may carry either. I hit exactly that, in the direction it warns about, and it nearly ended the investigation.

I queried .metadata["gc.work_dir"] on ce-91c, got null, and concluded the bead was clean while the controller kept naming it. The real key on that bead was the unprefixed work_dir. lu-zmn carried both, set to the same path. So a bead can hold either, or both, and checking one spelling produces a confident wrong answer that contradicts the log and makes the log look like the thing that is broken.

Worth promoting from a trap in the workaround to a line in the fix: whatever reads this should read both spellings, and whatever reports it should print which one it found.

A finding that ties this to gascity#1, and strengthens suggested fix 3

The issue proposes, as its third fix, dropping the unverifiable work_dir and allocating a fresh slot, on the grounds that refusing the whole trigger "protects nothing and stops the city." Tonight's case is stronger than that. Refusing the trigger does not merely fail to protect. It preserves a pointer to a worktree that would poison the next thing to use it.

ce-91c's stale work_dir pointed at gastown.nux-ce91c. That worktree is mid-rebase with unresolved conflicts:

UU apps/celilo/src/hooks/executor.ts
UU apps/celilo/src/hooks/jail.ts
UU openspec/changes/hook-process-boundary/tasks.md
HEAD detached at 8488bdde, stash@{0}: autostash

That is gascity#1 (a killed polecat's rebase state surviving in a reused worktree and VOIDing every future merge gate there). "Tree mid-rebase" is one of the named causes of a VOID verdict, so a polecat that had honoured that work_dir would have produced no verdict and held the bead.

So the two issues compound. #7 stalls the pool, and the evidence #7 is protecting is precisely the evidence #1 says is poisonous. Dropping the unverifiable work_dir and allocating fresh is not a convenience. On this bead it was the only outcome that could have produced a gate verdict at all.

I checked the committed work before accepting that, per the issue's second trap: polecat/ce-91c-rebased is on origin at 8da19330, the sha the bead names, three commits ahead of main. Nothing committed was lost. Only the abandoned rebase went, and the bead's own hold reason already said it needed a fresh rebase onto current main.

One detail from that branch's history that says how routine this is. Its third commit is e9cb483a wip(hooks): salvage uncommitted 4.7/4.8/4.12 work from dead rictus session. So this same bead had already survived one polecat death and a hand salvage before tonight's.

On fix 2, from the operator's chair

The issue says a skipped trigger is indistinguishable from no demand at every surface an operator uses, and that is exactly how it played out. The operator asked me to look at the polecats because he had noticed, by eye, that they had not picked up work "for a long time." gc status showed five stopped slots and no error. Nothing escalated. Nothing alerted. The daily digest for 2026-09-01 recorded "no merges to main in any rig" and a separate 24-hour pool stall (ce-jl0, four workers dead mid-claim) without connecting them to a cause.

A human noticing an absence is the current detection mechanism, and an absence is the hardest thing for a person to notice.

Mayor. This recurred within fourteen minutes of the issue being filed, which I think is the most important thing to add: **the workaround is a treadmill, not a fix.** ## It came back at 06:27Z, on two fresh beads This issue was filed at 06:13:06Z with five beads cleared and the pools spawning again. At 06:27Z both pools were stalled again, with `poolDesired = 1` and zero sessions, on two beads that were not in that first sweep: ``` buildDesiredState: pool "celilo/gastown.polecat" request: pool trigger worktree evidence invalid: work bead ce-91c has gc.work_dir=".../worktrees/celilo/polecats/gastown.nux-ce91c" but is missing gc.worktree_repo (skipping) buildDesiredState: pool "lunacycle/gastown.polecat" request: pool trigger worktree evidence invalid: work bead lu-zmn has gc.work_dir=".../worktrees/lunacycle/polecats/gastown.furiosa" but is missing gc.worktree_repo (skipping) ``` That follows from the issue's own mechanism and is worth stating out loud: a polecat writes `work_dir` on every claim, so **every claim re-arms the trap.** Clearing is not remediation that holds. It has to be re-run for the rest of the city's life, and each round is an outage that lasts until an operator happens to read a log they were never told about. This one ran from roughly 23:15 local until 06:27, so about seven hours, while three P1 beads sat routed and claimable. Same workaround, same result, verified the same way. Both pools spawned within 50 seconds of clearing (`celilo/gastown.furiosa` and `lunacycle/gastown.nux` running), and `lu-zmn` went to `in_progress` under `lunacycle/gastown.nux`. ## Confirming trap 1 the hard way The issue's first trap says to clear both key spellings because a bead may carry either. I hit exactly that, in the direction it warns about, and it nearly ended the investigation. I queried `.metadata["gc.work_dir"]` on `ce-91c`, got `null`, and concluded the bead was clean while the controller kept naming it. The real key on that bead was the unprefixed `work_dir`. `lu-zmn` carried **both**, set to the same path. So a bead can hold either, or both, and checking one spelling produces a confident wrong answer that contradicts the log and makes the log look like the thing that is broken. Worth promoting from a trap in the workaround to a line in the fix: whatever reads this should read both spellings, and whatever reports it should print which one it found. ## A finding that ties this to gascity#1, and strengthens suggested fix 3 The issue proposes, as its third fix, dropping the unverifiable `work_dir` and allocating a fresh slot, on the grounds that refusing the whole trigger "protects nothing and stops the city." Tonight's case is stronger than that. Refusing the trigger does not merely fail to protect. It **preserves a pointer to a worktree that would poison the next thing to use it.** `ce-91c`'s stale `work_dir` pointed at `gastown.nux-ce91c`. That worktree is mid-rebase with unresolved conflicts: ``` UU apps/celilo/src/hooks/executor.ts UU apps/celilo/src/hooks/jail.ts UU openspec/changes/hook-process-boundary/tasks.md HEAD detached at 8488bdde, stash@{0}: autostash ``` That is gascity#1 (a killed polecat's rebase state surviving in a reused worktree and VOIDing every future merge gate there). "Tree mid-rebase" is one of the named causes of a VOID verdict, so a polecat that had honoured that `work_dir` would have produced no verdict and held the bead. So the two issues compound. #7 stalls the pool, and the evidence #7 is protecting is precisely the evidence #1 says is poisonous. Dropping the unverifiable `work_dir` and allocating fresh is not a convenience. On this bead it was the only outcome that could have produced a gate verdict at all. I checked the committed work before accepting that, per the issue's second trap: `polecat/ce-91c-rebased` is on origin at `8da19330`, the sha the bead names, three commits ahead of main. Nothing committed was lost. Only the abandoned rebase went, and the bead's own hold reason already said it needed a fresh rebase onto current main. One detail from that branch's history that says how routine this is. Its third commit is `e9cb483a wip(hooks): salvage uncommitted 4.7/4.8/4.12 work from dead rictus session`. So this same bead had already survived one polecat death and a hand salvage before tonight's. ## On fix 2, from the operator's chair The issue says a skipped trigger is indistinguishable from no demand at every surface an operator uses, and that is exactly how it played out. The operator asked me to look at the polecats because he had noticed, by eye, that they had not picked up work "for a long time." `gc status` showed five stopped slots and no error. Nothing escalated. Nothing alerted. The daily digest for 2026-09-01 recorded "no merges to main in any rig" and a separate 24-hour pool stall (`ce-jl0`, four workers dead mid-claim) without connecting them to a cause. A human noticing an absence is the current detection mechanism, and an absence is the hardest thing for a person to notice.
Author
Owner

A second variant, and it is not missing keys — the format is wrong

Measured 2026-09-09 on gc-scratch, celilo pool. Same function, same blast radius, different mechanism, and the existing band-aid does not catch it.

This issue describes evidence that is incomplete: work_dir present, one of the nine provenance keys absent, written by a pack that publishes none of them. This one had all nine keys present, and was still rejected:

buildDesiredState: pool "celilo/gastown.polecat" request: pool trigger worktree
evidence: verification failed: verifying worktree
".../worktrees/celilo/ce-0f5p" provenance: store ref is "celilo",
want "rig:celilo" (skipping)

The producer and the validator disagree on the value format of the store ref, not on whether the key exists. celilo versus rig:celilo.

And the producer is not worktree-setup.sh. The bead carried gc.worktree_creator=mol-polecat-work along with a complete set:

gc.worktree_repo        gc.worktree_root         gc.worktree_base_ref
gc.worktree_base_sha    gc.worktree_creator      gc.worktree_owner
gc.worktree_generation  gc.worktree_lifecycle

So there are at least two producers of worktree provenance that the binary rejects, for two different reasons. Fix suggestion 1 above ("make the pack and the binary agree") is right and is bigger than one script.

Same blast radius: one stale bead stops the whole pool

The rejected trigger was ce-0f5p, an open unassigned bead nobody was working. The pool request as a whole was skipped, so an unrelated bead (ce-ko64, P2, correctly routed, in gc bd ready, no work_dir of its own) sat unclaimed for 60 minutes while two of five slots idled.

poolDesired read 4 and three sessions ran, every tick, 59 ticks. Nothing else said anything.

unstick-pool-triggers did not clear it

The 2-minute order that exists as the band-aid for this issue ran throughout that hour and did not unstick it. Whatever it matches on, it does not match this shape. Worth checking before anyone relies on it as coverage.

Workaround, and one correction to the one above

The documented workaround uses --unset-metadata. I used empty values, and it worked:

gc bd update ce-0f5p --set-metadata work_dir= --set-metadata gc.work_dir=

Pool went from 3 running to 5 within one minute, and the skip lines stopped. Verified by the sessions existing, not by the exit code.

Extend trap 2: git diff is the wrong instrument

This issue already warns to check the old worktree for uncommitted work. It cost me nearly more than that, because git diff does not show untracked files.

ce-0f5p's worktree held five modified files, which git diff showed, plus an untracked 289-line test file it did not. That file is plausibly the standing gate celilo#1327 asks for, and it was the single most valuable thing in the directory. Had I captured only the diff and cleared the evidence, it would have been silently orphaned on the next claim.

Use git status --porcelain and commit everything, not git diff. The work was recovered to origin/polecat/ce-0f5p (commit 95b00a5a, 6 files, +332) before clearing.

Root cause of the orphan itself is #18: nux was drained mid-claim and released the bead with work still on disk.

## A second variant, and it is not missing keys — the format is wrong Measured 2026-09-09 on gc-scratch, celilo pool. Same function, same blast radius, different mechanism, and the existing band-aid does not catch it. This issue describes evidence that is **incomplete**: `work_dir` present, one of the nine provenance keys absent, written by a pack that publishes none of them. This one had **all nine keys present**, and was still rejected: ``` buildDesiredState: pool "celilo/gastown.polecat" request: pool trigger worktree evidence: verification failed: verifying worktree ".../worktrees/celilo/ce-0f5p" provenance: store ref is "celilo", want "rig:celilo" (skipping) ``` The producer and the validator disagree on the **value format** of the store ref, not on whether the key exists. `celilo` versus `rig:celilo`. And the producer is not `worktree-setup.sh`. The bead carried `gc.worktree_creator=mol-polecat-work` along with a complete set: ``` gc.worktree_repo gc.worktree_root gc.worktree_base_ref gc.worktree_base_sha gc.worktree_creator gc.worktree_owner gc.worktree_generation gc.worktree_lifecycle ``` So there are at least **two** producers of worktree provenance that the binary rejects, for two different reasons. Fix suggestion 1 above ("make the pack and the binary agree") is right and is bigger than one script. ## Same blast radius: one stale bead stops the whole pool The rejected trigger was `ce-0f5p`, an open unassigned bead nobody was working. The pool request as a whole was skipped, so an **unrelated** bead (`ce-ko64`, P2, correctly routed, in `gc bd ready`, no `work_dir` of its own) sat unclaimed for **60 minutes** while two of five slots idled. `poolDesired` read 4 and three sessions ran, every tick, 59 ticks. Nothing else said anything. ## `unstick-pool-triggers` did not clear it The 2-minute order that exists as the band-aid for this issue ran throughout that hour and did not unstick it. Whatever it matches on, it does not match this shape. Worth checking before anyone relies on it as coverage. ## Workaround, and one correction to the one above The documented workaround uses `--unset-metadata`. I used empty values, and it worked: ``` gc bd update ce-0f5p --set-metadata work_dir= --set-metadata gc.work_dir= ``` Pool went from 3 running to 5 within one minute, and the skip lines stopped. Verified by the sessions existing, not by the exit code. ## Extend trap 2: `git diff` is the wrong instrument This issue already warns to check the old worktree for uncommitted work. It cost me nearly more than that, because `git diff` **does not show untracked files**. `ce-0f5p`'s worktree held five modified files, which `git diff` showed, **plus an untracked 289-line test file it did not**. That file is plausibly the standing gate celilo#1327 asks for, and it was the single most valuable thing in the directory. Had I captured only the diff and cleared the evidence, it would have been silently orphaned on the next claim. Use `git status --porcelain` and commit everything, not `git diff`. The work was recovered to `origin/polecat/ce-0f5p` (commit `95b00a5a`, 6 files, +332) before clearing. Root cause of the orphan itself is #18: nux was drained mid-claim and released the bead with work still on disk.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
celilo/gascity#7
No description provided.