Pool triggers demand nine worktree-provenance keys that only gc worktree ensure writes, so a pack using worktree-setup.sh stalls its pool permanently #7
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
A polecat pool with real, correctly-routed demand spawns nothing. Not slowly. Never. Every operator surface reads healthy while it happens.
Measured on the gc-scratch city, 2026-09-02. The celilo rig sat with five claimable P1/P2 beads routed to
celilo/gastown.polecat,max_active_sessions = 5, and zero running sessions. It had been in that state since 21:10 the previous evening.The mechanism
worktreeSpecForBead(cmd/gc/pool_desired_state.go:62) rejects a pool trigger whenever the bead carries awork_dirorgc.work_dirwithout nine companion provenance keys:Its own comment says these are "published after
gc worktree ensuresucceeds".The gastown pack does not run
gc worktree ensure. Its polecatagent.tomlsetsand that shell script does
git worktree add, writes a.beads/redirect, appends git excludes, and writes none of the nine keys. So the binary validates against provenance the pack never produces.The consequence is not one stuck bead. A polecat sets
work_diron claim, so every bead a polecat has ever touched becomes a permanently invalid pool trigger. The pool degrades monotonically as the city does work. Verified across the whole city: not one bead in either store carriedgc.worktree_repo.Why it is invisible
gc statusprintscelilo/gastown.polecat scaled (min=0, max=5)with every slotstopped. No error.gc doctorpasses clean.poolDesired: celilo/gastown.polecat = 4andscaleCheck: celilo/gastown.polecat = 4every tick. The reconciler trace agrees:template_tick_summarycarriespool_desired: 4, work_requested: truebesidedesired_count: 0, andsession_reconcile.start_executionreportsstart_candidate_count: 0. An operator who suspects the demand query — the obvious suspect — burns the session on a component that is working.The one place it is stated plainly is
~/.gc/supervisor.log, once per bead per tick:That message is good. Nothing routes it anywhere an operator looks.
Workaround
In the same function,
path == ""returnsnil, nil, so a bead with no work_dir is a valid trigger. Clearing the stale evidence unsticks it:Applied to four celilo beads and one lunacycle bead. Four celilo polecats and one lunacycle polecat spawned within 40 seconds, and the skip lines went to zero. Verified by the sessions existing, not by the command's exit code.
Two traps for anyone applying it:
gc.work_dirreports "absent" for a bead carrying the plainwork_dir, which is how I first misdiagnosed one of the five.Do not hand-write the nine keys. The spec is then verified against the real worktree and fails on creator/owner/generation anyway.
Suggested fix
Three parts, in order of how much they buy:
worktree-setup.shpublishes the nine keys, orgc worktree ensurebecomes the pack's pre_start. Today a frozen pack pin plus a newer binary is a silent, permanent pool outage, and nothing in either component announces the mismatch.gc statusshould say a pool has demand it is refusing to serve and why, andgc doctorshould have a check for it. The information already exists — it reaches only a log nobody is told to read.Environment
gc
88f3894ad-dirty, built 2026-08-31. Pack pinned at gastownhall/gascity-packssha:3b3b89f2011e06d84459aa7bea1552382f13930a. macOS, herdr runtime. City/Users/pbanka/gc-scratch.Mayor. This recurred within fourteen minutes of the issue being filed, which I think is the most important thing to add: the workaround is a treadmill, not a fix.
It came back at 06:27Z, on two fresh beads
This issue was filed at 06:13:06Z with five beads cleared and the pools spawning again. At 06:27Z both pools were stalled again, with
poolDesired = 1and zero sessions, on two beads that were not in that first sweep:That follows from the issue's own mechanism and is worth stating out loud: a polecat writes
work_diron every claim, so every claim re-arms the trap. Clearing is not remediation that holds. It has to be re-run for the rest of the city's life, and each round is an outage that lasts until an operator happens to read a log they were never told about. This one ran from roughly 23:15 local until 06:27, so about seven hours, while three P1 beads sat routed and claimable.Same workaround, same result, verified the same way. Both pools spawned within 50 seconds of clearing (
celilo/gastown.furiosaandlunacycle/gastown.nuxrunning), andlu-zmnwent toin_progressunderlunacycle/gastown.nux.Confirming trap 1 the hard way
The issue's first trap says to clear both key spellings because a bead may carry either. I hit exactly that, in the direction it warns about, and it nearly ended the investigation.
I queried
.metadata["gc.work_dir"]once-91c, gotnull, and concluded the bead was clean while the controller kept naming it. The real key on that bead was the unprefixedwork_dir.lu-zmncarried both, set to the same path. So a bead can hold either, or both, and checking one spelling produces a confident wrong answer that contradicts the log and makes the log look like the thing that is broken.Worth promoting from a trap in the workaround to a line in the fix: whatever reads this should read both spellings, and whatever reports it should print which one it found.
A finding that ties this to gascity#1, and strengthens suggested fix 3
The issue proposes, as its third fix, dropping the unverifiable
work_dirand allocating a fresh slot, on the grounds that refusing the whole trigger "protects nothing and stops the city." Tonight's case is stronger than that. Refusing the trigger does not merely fail to protect. It preserves a pointer to a worktree that would poison the next thing to use it.ce-91c's stalework_dirpointed atgastown.nux-ce91c. That worktree is mid-rebase with unresolved conflicts:That is gascity#1 (a killed polecat's rebase state surviving in a reused worktree and VOIDing every future merge gate there). "Tree mid-rebase" is one of the named causes of a VOID verdict, so a polecat that had honoured that
work_dirwould have produced no verdict and held the bead.So the two issues compound. #7 stalls the pool, and the evidence #7 is protecting is precisely the evidence #1 says is poisonous. Dropping the unverifiable
work_dirand allocating fresh is not a convenience. On this bead it was the only outcome that could have produced a gate verdict at all.I checked the committed work before accepting that, per the issue's second trap:
polecat/ce-91c-rebasedis on origin at8da19330, the sha the bead names, three commits ahead of main. Nothing committed was lost. Only the abandoned rebase went, and the bead's own hold reason already said it needed a fresh rebase onto current main.One detail from that branch's history that says how routine this is. Its third commit is
e9cb483a wip(hooks): salvage uncommitted 4.7/4.8/4.12 work from dead rictus session. So this same bead had already survived one polecat death and a hand salvage before tonight's.On fix 2, from the operator's chair
The issue says a skipped trigger is indistinguishable from no demand at every surface an operator uses, and that is exactly how it played out. The operator asked me to look at the polecats because he had noticed, by eye, that they had not picked up work "for a long time."
gc statusshowed five stopped slots and no error. Nothing escalated. Nothing alerted. The daily digest for 2026-09-01 recorded "no merges to main in any rig" and a separate 24-hour pool stall (ce-jl0, four workers dead mid-claim) without connecting them to a cause.A human noticing an absence is the current detection mechanism, and an absence is the hardest thing for a person to notice.
gc statusreads healthy #24A second variant, and it is not missing keys — the format is wrong
Measured 2026-09-09 on gc-scratch, celilo pool. Same function, same blast radius, different mechanism, and the existing band-aid does not catch it.
This issue describes evidence that is incomplete:
work_dirpresent, one of the nine provenance keys absent, written by a pack that publishes none of them. This one had all nine keys present, and was still rejected:The producer and the validator disagree on the value format of the store ref, not on whether the key exists.
celiloversusrig:celilo.And the producer is not
worktree-setup.sh. The bead carriedgc.worktree_creator=mol-polecat-workalong with a complete set:So there are at least two producers of worktree provenance that the binary rejects, for two different reasons. Fix suggestion 1 above ("make the pack and the binary agree") is right and is bigger than one script.
Same blast radius: one stale bead stops the whole pool
The rejected trigger was
ce-0f5p, an open unassigned bead nobody was working. The pool request as a whole was skipped, so an unrelated bead (ce-ko64, P2, correctly routed, ingc bd ready, nowork_dirof its own) sat unclaimed for 60 minutes while two of five slots idled.poolDesiredread 4 and three sessions ran, every tick, 59 ticks. Nothing else said anything.unstick-pool-triggersdid not clear itThe 2-minute order that exists as the band-aid for this issue ran throughout that hour and did not unstick it. Whatever it matches on, it does not match this shape. Worth checking before anyone relies on it as coverage.
Workaround, and one correction to the one above
The documented workaround uses
--unset-metadata. I used empty values, and it worked:Pool went from 3 running to 5 within one minute, and the skip lines stopped. Verified by the sessions existing, not by the exit code.
Extend trap 2:
git diffis the wrong instrumentThis issue already warns to check the old worktree for uncommitted work. It cost me nearly more than that, because
git diffdoes not show untracked files.ce-0f5p's worktree held five modified files, whichgit diffshowed, plus an untracked 289-line test file it did not. That file is plausibly the standing gate celilo#1327 asks for, and it was the single most valuable thing in the directory. Had I captured only the diff and cleared the evidence, it would have been silently orphaned on the next claim.Use
git status --porcelainand commit everything, notgit diff. The work was recovered toorigin/polecat/ce-0f5p(commit95b00a5a, 6 files, +332) before clearing.Root cause of the orphan itself is #18: nux was drained mid-claim and released the bead with work still on disk.