celilo/gascity

Fork 0

Supervisor drains a polecat within a minute of its claim: assigned work is read from the beads cache, demand from the store, so a fresh claim is invisible to one and already gone from the other for one reconcile cadence #18

New issue

Open

opened 2026-09-06 10:45:18 +00:00 by forgejo-admin · 6 comments

forgejo-admin commented

2026-09-06 10:45:18 +00:00

Owner

What happened

gc-scratch, 2026-09-06 10:41Z. The celilo polecat pool had two sessions with work (furiosa on ce-013r, nux on ce-1h2o) and one unclaimed routed bead, ce-ilt6. The supervisor spawned celilo--gastown__rictus for it (start outcome=success at 10:41:0x). rictus claimed ce-ilt6. Within the same minute the supervisor drained rictus with no-wake-reason, the claim was released, and ce-ilt6 went back to open and unassigned with rictus's work_dir and gc.session_name left on it.

The raw lines from ~/.gc/supervisor.log, in order, with the routine assignee rows for named sessions removed:

poolDesired: celilo/gastown.polecat = 3
scaleCheck: celilo/gastown.polecat = 1
2026/09/06 03:41:05 api: GET /v0/city/gc-scratch/beads 200
assignedWorkBeads: 12 beads found
  ce-013r assignee=celilo/gastown.furiosa routed=celilo/gastown.polecat status=in_progress
  ce-1h2o assignee=celilo/gastown.nux routed=celilo/gastown.polecat status=in_progress
  ce-jom4 assignee=gs-yxywi routed= status=open
2026/09/06 03:41:20 beads cache: reconciled rig=ce beads=148 adds=0 updates=1 removes=0 took=44ms cadence=default
namedWorkReady: 12 assigned beads, 7 named specs, ...
poolDesired: celilo/gastown.polecat = 2
Draining session 'celilo--gastown__rictus': no-wake-reason

Read it as two queries in one tick that disagree:

assignedWorkBeads lists only two in-progress pool beads. rictus's claim on ce-ilt6 is not there. The updates=1 in the cache reconcile that follows is that claim arriving.
The demand side (the bd ready ... --unassigned probe against the store) had already stopped counting ce-ilt6, because at the store it was assigned.

So for one reconcile cadence the claim is invisible to the instrument that counts sessions with work and already gone from the instrument that counts demand. poolDesired comes out as 2 with three sessions running, and the supervisor drains the one it cannot attribute work to, which is exactly the one that just claimed.

How often

Tonight, three fresh polecat claims in a row: two survived (their claims happened to land in the cache before the next poolDesired), one was drained. The same session name was drained the same way once earlier (celilo--gastown__rictus on ce-ymot at 10:07Z, after an hour of work, with a committed and unpushed fix in its worktree that the mayor had to recover by hand). Across the last 200k log lines, no-wake-reason drains: rictus 4, furiosa 2, nux 1, slit 1.

Cost

Every hit costs a full polecat start (about 15s plus a worktree and a provider session), the polecat's context, and in the ce-ymot case nearly a lost fix. The bead is left carrying work_dir and gc.session_name from the dead claim, which is the input to celilo/gascity#7's evidence check, so the two defects feed each other: the drained claim re-arms #7 on the next spawn.

What would fix it

Compute poolDesired from one snapshot. Either read both the assigned set and the demand set from the beads cache, or both from the store, in the same tick. Or treat a session younger than one reconcile cadence as having work until the cache catches up. A drain decision should never rest on two reads taken across a reconcile boundary.

#7 is the worktree-evidence validation on pool triggers. This one fires with no evidence invalid line in the tick that drains. Recorded separately so a fix for #7 is not read as covering it.

## What happened gc-scratch, 2026-09-06 10:41Z. The celilo polecat pool had two sessions with work (furiosa on ce-013r, nux on ce-1h2o) and one unclaimed routed bead, ce-ilt6. The supervisor spawned `celilo--gastown__rictus` for it (start outcome=success at 10:41:0x). rictus claimed ce-ilt6. Within the same minute the supervisor drained rictus with `no-wake-reason`, the claim was released, and ce-ilt6 went back to open and unassigned with rictus's `work_dir` and `gc.session_name` left on it. The raw lines from `~/.gc/supervisor.log`, in order, with the routine assignee rows for named sessions removed: ``` poolDesired: celilo/gastown.polecat = 3 scaleCheck: celilo/gastown.polecat = 1 2026/09/06 03:41:05 api: GET /v0/city/gc-scratch/beads 200 assignedWorkBeads: 12 beads found ce-013r assignee=celilo/gastown.furiosa routed=celilo/gastown.polecat status=in_progress ce-1h2o assignee=celilo/gastown.nux routed=celilo/gastown.polecat status=in_progress ce-jom4 assignee=gs-yxywi routed= status=open 2026/09/06 03:41:20 beads cache: reconciled rig=ce beads=148 adds=0 updates=1 removes=0 took=44ms cadence=default namedWorkReady: 12 assigned beads, 7 named specs, ... poolDesired: celilo/gastown.polecat = 2 Draining session 'celilo--gastown__rictus': no-wake-reason ``` Read it as two queries in one tick that disagree: - `assignedWorkBeads` lists only two in-progress pool beads. rictus's claim on ce-ilt6 is not there. The `updates=1` in the cache reconcile that follows is that claim arriving. - The demand side (the `bd ready ... --unassigned` probe against the store) had already stopped counting ce-ilt6, because at the store it was assigned. So for one reconcile cadence the claim is invisible to the instrument that counts sessions with work and already gone from the instrument that counts demand. `poolDesired` comes out as 2 with three sessions running, and the supervisor drains the one it cannot attribute work to, which is exactly the one that just claimed. ## How often Tonight, three fresh polecat claims in a row: two survived (their claims happened to land in the cache before the next `poolDesired`), one was drained. The same session name was drained the same way once earlier (`celilo--gastown__rictus` on ce-ymot at 10:07Z, after an hour of work, with a committed and unpushed fix in its worktree that the mayor had to recover by hand). Across the last 200k log lines, `no-wake-reason` drains: rictus 4, furiosa 2, nux 1, slit 1. ## Cost Every hit costs a full polecat start (about 15s plus a worktree and a provider session), the polecat's context, and in the ce-ymot case nearly a lost fix. The bead is left carrying `work_dir` and `gc.session_name` from the dead claim, which is the input to celilo/gascity#7's evidence check, so the two defects feed each other: the drained claim re-arms #7 on the next spawn. ## What would fix it Compute `poolDesired` from one snapshot. Either read both the assigned set and the demand set from the beads cache, or both from the store, in the same tick. Or treat a session younger than one reconcile cadence as having work until the cache catches up. A drain decision should never rest on two reads taken across a reconcile boundary. ## Not related to #7 #7 is the worktree-evidence validation on pool triggers. This one fires with no `evidence invalid` line in the tick that drains. Recorded separately so a fix for #7 is not read as covering it.

forgejo-admin commented

2026-09-06 11:08:07 +00:00

Author

Owner

Update, 11:10Z: it is not only at claim time. A successful spawn drains live neighbours, and the trigger-bead mapping is wrong.

The spawn and no-wake-reason drain events for the celilo polecat pool tonight, in log order (~/.gc/supervisor.log, line numbers):

1181940 START furiosa
1182155 DRAIN furiosa          (its own start, then drained; respawned at 1183040 and survived)
1183723 START nux              (ce-1h2o)
1184712 START rictus           (ce-ilt6)
1184788 DRAIN rictus           (the case in the issue body)
1185493 START rictus           (survived, finished ce-ilt6)
1186625 START capable          (ce-kg5o, a hold lifted by the mayor)
1186626 DRAIN nux              (mid-work on ce-1h2o for 30 min, a --keep e2e stack left up with a dead runner pid)
1186681 DRAIN furiosa          (mid-work on ce-013r for 35 min)
1187259 START furiosa          (respawn; kept ce-013r because the assignee is the agent name)
1188201 START nux              (respawn; ce-1h2o's claim had been released, so it starts cold)

Four spawns, three followed within a tick by a drain. At the 11:03Z tick the assignedWorkBeads block listed both ce-1h2o assignee=celilo/gastown.nux status=in_progress and ce-013r assignee=celilo/gastown.furiosa status=in_progress, so the cache-lag explanation in the body does not cover this one. What the log shows instead:

buildDesiredState: pool "celilo/gastown.polecat" session gs-mguyjz trigger bead ce-1h2o: pool trigger worktree evidence invalid: work bead ce-1h2o has gc.work_dir="..." but is missing gc.worktree_repo (skipping)

gs-mguyjz is capable, which held ce-kg5o. The supervisor is validating worktree evidence against a trigger bead that is not the session's claim, and the #7 rejection then removes a live session from the desired set. So #7 and this issue are one mechanism seen from two sides: #7 blocks a spawn when the trigger bead carries a bare work_dir; here the same check, applied to a running session with a mismapped trigger, drains it.

Whether the bead survives is a coin flip. furiosa kept ce-013r because the assignee is the agent name and the respawned session matched it. nux's claim on ce-1h2o was released, and its kept e2e stack (nine containers) stayed up under cele2e's flock with a dead runner pid, which is the stranded-rig shape from a new door.

Operating consequence recorded in gc-scratch CLAUDE.md: lifting a hold is not free while polecats are working, because the spawn it causes may kill one of them.

**Update, 11:10Z: it is not only at claim time. A successful spawn drains live neighbours, and the trigger-bead mapping is wrong.** The spawn and no-wake-reason drain events for the celilo polecat pool tonight, in log order (`~/.gc/supervisor.log`, line numbers): ``` 1181940 START furiosa 1182155 DRAIN furiosa (its own start, then drained; respawned at 1183040 and survived) 1183723 START nux (ce-1h2o) 1184712 START rictus (ce-ilt6) 1184788 DRAIN rictus (the case in the issue body) 1185493 START rictus (survived, finished ce-ilt6) 1186625 START capable (ce-kg5o, a hold lifted by the mayor) 1186626 DRAIN nux (mid-work on ce-1h2o for 30 min, a --keep e2e stack left up with a dead runner pid) 1186681 DRAIN furiosa (mid-work on ce-013r for 35 min) 1187259 START furiosa (respawn; kept ce-013r because the assignee is the agent name) 1188201 START nux (respawn; ce-1h2o's claim had been released, so it starts cold) ``` Four spawns, three followed within a tick by a drain. At the 11:03Z tick the `assignedWorkBeads` block listed both `ce-1h2o assignee=celilo/gastown.nux status=in_progress` and `ce-013r assignee=celilo/gastown.furiosa status=in_progress`, so the cache-lag explanation in the body does not cover this one. What the log shows instead: ``` buildDesiredState: pool "celilo/gastown.polecat" session gs-mguyjz trigger bead ce-1h2o: pool trigger worktree evidence invalid: work bead ce-1h2o has gc.work_dir="..." but is missing gc.worktree_repo (skipping) ``` `gs-mguyjz` is capable, which held ce-kg5o. The supervisor is validating worktree evidence against a trigger bead that is not the session's claim, and the #7 rejection then removes a live session from the desired set. So #7 and this issue are one mechanism seen from two sides: #7 blocks a spawn when the trigger bead carries a bare `work_dir`; here the same check, applied to a running session with a mismapped trigger, drains it. Whether the bead survives is a coin flip. furiosa kept ce-013r because the assignee is the agent name and the respawned session matched it. nux's claim on ce-1h2o was released, and its kept e2e stack (nine containers) stayed up under cele2e's flock with a dead runner pid, which is the stranded-rig shape from a new door. Operating consequence recorded in gc-scratch CLAUDE.md: lifting a hold is not free while polecats are working, because the spawn it causes may kill one of them.

forgejo-admin commented

2026-09-06 12:59:10 +00:00

Author

Owner

12:50Z to 12:58Z, a batch into an empty pool. Three beads lifted at once into a pool with zero sessions. Three polecats spawned (nux, rictus, slit). nux was drained no-wake-reason before it claimed anything and respawned. rictus and slit spent five minutes reading (slow provider) without claiming. The respawned nux claimed ce-cbr3 at 12:57Z, and on the next tick both rictus and slit were drained no-wake-reason, still unclaimed, with four routed unheld beads available to them and poolDesired reading 2 and 3 in alternation.

So the drain fires on any pool membership change, not only on a spawn: a neighbour's claim is enough. Sessions that have not yet claimed are drained as surplus even while demand exceeds the running count. gc-scratch is now operating the celilo pool at size one until this is fixed.

**12:50Z to 12:58Z, a batch into an empty pool.** Three beads lifted at once into a pool with zero sessions. Three polecats spawned (nux, rictus, slit). nux was drained `no-wake-reason` before it claimed anything and respawned. rictus and slit spent five minutes reading (slow provider) without claiming. The respawned nux claimed ce-cbr3 at 12:57Z, and on the next tick both rictus and slit were drained `no-wake-reason`, still unclaimed, with four routed unheld beads available to them and `poolDesired` reading 2 and 3 in alternation. So the drain fires on any pool membership change, not only on a spawn: a neighbour's claim is enough. Sessions that have not yet claimed are drained as surplus even while demand exceeds the running count. gc-scratch is now operating the celilo pool at size one until this is fixed.

forgejo-admin commented

2026-09-06 18:00:46 +00:00

Author

Owner

Another instance, measured 2026-09-06T17:53Z to 17:58Z by the celilo refinery, with an unusually clean trigger.

A mistaken gc sling celilo/gastown.polecat 0 (the literal text 0, which sling turns into an auto-created task bead) created ce-y1e4, open, routed, unassigned. Supervisor log, in order:

session lifecycle: op=start wave=0 session=celilo--gastown__rictus template=celilo/gastown.polecat outcome=start_enqueued
Auto-closed convoy ce-8es4 "sling-ce-y1e4"
Woke session 'celilo/gastown.rictus'
session lifecycle: op=start ... session=celilo--gastown__rictus ... outcome=success duration=15.141s
Draining session 'celilo--gastown__furiosa': no-wake-reason

furiosa was mid-suite on ce-21hp (aspect-fanout-new-systems, --keep, holding the city's e2e lock). Its claim was released: ce-21hp went back to open, unassigned, routed, with gc.e2e_lock=held still on it, while the cele2e run lock stayed BUSY on furiosa's kept stack (pid 25007, heartbeat live). So this is the "claim released" side of the coin flip the issue describes, and the trigger bead was junk that was closed two minutes after creation. One spawn, one drained worker, one orphaned kept stack.

The refinery put hold:mayor on ce-21hp to stop the released claim reading as demand for a second spawn (which would have drained nux).

Another instance, measured 2026-09-06T17:53Z to 17:58Z by the celilo refinery, with an unusually clean trigger. A mistaken `gc sling celilo/gastown.polecat 0` (the literal text `0`, which sling turns into an auto-created task bead) created ce-y1e4, open, routed, unassigned. Supervisor log, in order: ``` session lifecycle: op=start wave=0 session=celilo--gastown__rictus template=celilo/gastown.polecat outcome=start_enqueued Auto-closed convoy ce-8es4 "sling-ce-y1e4" Woke session 'celilo/gastown.rictus' session lifecycle: op=start ... session=celilo--gastown__rictus ... outcome=success duration=15.141s Draining session 'celilo--gastown__furiosa': no-wake-reason ``` furiosa was mid-suite on ce-21hp (aspect-fanout-new-systems, `--keep`, holding the city's e2e lock). Its claim was released: ce-21hp went back to open, unassigned, routed, with `gc.e2e_lock=held` still on it, while the cele2e run lock stayed BUSY on furiosa's kept stack (pid 25007, heartbeat live). So this is the "claim released" side of the coin flip the issue describes, and the trigger bead was junk that was closed two minutes after creation. One spawn, one drained worker, one orphaned kept stack. The refinery put `hold:mayor` on ce-21hp to stop the released claim reading as demand for a second spawn (which would have drained nux).

forgejo-admin referenced this issue

2026-09-06 18:13:01 +00:00

A fresh polecat spawn reuses a worktree that is still another bead's refinery-held work_dir, so the formula's branch-setup would reset the checkout under the running merge gate #19

forgejo-admin commented

2026-09-07 20:16:49 +00:00

Author

Owner

2026-09-07 19:55Z: the shed picked the session that HAD the claim, and kept the one that did not.

This one does not fit either explanation already on the issue. The body blames cache lag, where the fresh claim is invisible. Comment 6988 blames a trigger bead mismapped to the wrong session. Here the bookkeeping was correct in the drain tick itself, and the supervisor still drained the working session.

The sequence from ~/.gc/supervisor.log, routine rows removed:

12:55:05  session lifecycle: op=start session=celilo--gastown__furiosa outcome=success duration=20.859s
          poolDesired: celilo/gastown.polecat = 2
          scaleCheck:  celilo/gastown.polecat = 1
12:55:21  beads cache: reconciled rig=gs beads=133 adds=1 updates=1 removes=0
          poolDesired: celilo/gastown.polecat = 2
          scaleCheck:  celilo/gastown.polecat = 1
          poolDesired: celilo/gastown.polecat = 2
          scaleCheck:  celilo/gastown.polecat = 1
12:55:41  beads cache: reconciled rig=ce beads=131 adds=0 updates=0 removes=0
          poolDesired: celilo/gastown.polecat = 1
          Draining session 'celilo--gastown__nux': no-wake-reason

The assignedWorkBeads block in the draining tick listed nux's claim correctly:

assignedWorkBeads: 13 beads found
  ce-5kg5 assignee=celilo/gastown.nux routed=celilo/gastown.polecat status=in_progress
  ...

So at the moment of the drain:

nux held ce-5kg5 in_progress, correctly routed, and the supervisor listed it.
furiosa was 36 seconds old, well inside postCreateProtectionTimeout (2m, cmd/gc/city_runtime.go:342), and had no claim in that tick's assignedWorkBeads at all.
poolDesired fell from 2 to 1 across a rig=ce reconcile reporting adds=0 updates=0 removes=0, so no bead in the celilo store changed between the tick that said 2 and the tick that said 1.
The shed took nux, the one with the visible claim. furiosa, the one with no claim, survived.

Why this matters more than the count being wrong. Even granting that poolDesired should have been 1, the choice of victim is the damaging part. A pool that must shed a session should shed one that is idle before one that holds in-progress work, and the resume tier already knows which is which: it is built from exactly the beads whose assignee resolves to a live session (computePoolDesiredStatesAt, cmd/gc/pool_desired_state.go:290). The information needed to pick correctly is computed in the same function that produces the number.

What it cost here. nux's session died with commit 10a32ed8 sitting unpushed in its worktree. The claim itself survived, because the assignee is the agent name celilo/gastown.nux and the respawn matched it, so this is the lucky side of the coin flip described in comment 6988. The worktree is per agent name, so the commit was still there afterwards. What was lost is the session and its context, plus a 20s restart. Had the operator not looked, the unpushed commit would have been one more drain away from a hand recovery, which is what happened to rictus on ce-ymot.

What I could not determine from the log, and I am not going to guess: why poolDesired dropped to 1. Nothing in the celilo store changed in that reconcile, and furiosa was inside its protection window, so neither the cache-lag path nor the protection-window expiry explains it on its face. recordReconcileTraceInputs (cmd/gc/city_runtime.go:2586) already captures desiredState, poolDesired and workSet per tick. Dumping that trace for one drain tick would settle both halves at once: what the count was built from, and which request the accepted slot named.

Filed from gc-scratch, which continues to operate the celilo polecat pool at size one to two for this reason.

**2026-09-07 19:55Z: the shed picked the session that HAD the claim, and kept the one that did not.** This one does not fit either explanation already on the issue. The body blames cache lag, where the fresh claim is invisible. Comment 6988 blames a trigger bead mismapped to the wrong session. Here the bookkeeping was correct in the drain tick itself, and the supervisor still drained the working session. The sequence from `~/.gc/supervisor.log`, routine rows removed: ``` 12:55:05 session lifecycle: op=start session=celilo--gastown__furiosa outcome=success duration=20.859s poolDesired: celilo/gastown.polecat = 2 scaleCheck: celilo/gastown.polecat = 1 12:55:21 beads cache: reconciled rig=gs beads=133 adds=1 updates=1 removes=0 poolDesired: celilo/gastown.polecat = 2 scaleCheck: celilo/gastown.polecat = 1 poolDesired: celilo/gastown.polecat = 2 scaleCheck: celilo/gastown.polecat = 1 12:55:41 beads cache: reconciled rig=ce beads=131 adds=0 updates=0 removes=0 poolDesired: celilo/gastown.polecat = 1 Draining session 'celilo--gastown__nux': no-wake-reason ``` The `assignedWorkBeads` block in the draining tick listed nux's claim correctly: ``` assignedWorkBeads: 13 beads found ce-5kg5 assignee=celilo/gastown.nux routed=celilo/gastown.polecat status=in_progress ... ``` So at the moment of the drain: - nux held `ce-5kg5` in_progress, correctly routed, and the supervisor listed it. - furiosa was 36 seconds old, well inside `postCreateProtectionTimeout` (2m, `cmd/gc/city_runtime.go:342`), and had no claim in that tick's `assignedWorkBeads` at all. - `poolDesired` fell from 2 to 1 across a `rig=ce` reconcile reporting `adds=0 updates=0 removes=0`, so no bead in the celilo store changed between the tick that said 2 and the tick that said 1. - The shed took **nux**, the one with the visible claim. **furiosa**, the one with no claim, survived. **Why this matters more than the count being wrong.** Even granting that `poolDesired` should have been 1, the choice of victim is the damaging part. A pool that must shed a session should shed one that is idle before one that holds in-progress work, and the resume tier already knows which is which: it is built from exactly the beads whose assignee resolves to a live session (`computePoolDesiredStatesAt`, `cmd/gc/pool_desired_state.go:290`). The information needed to pick correctly is computed in the same function that produces the number. **What it cost here.** nux's session died with commit `10a32ed8` sitting unpushed in its worktree. The claim itself survived, because the assignee is the agent name `celilo/gastown.nux` and the respawn matched it, so this is the lucky side of the coin flip described in comment 6988. The worktree is per agent name, so the commit was still there afterwards. What was lost is the session and its context, plus a 20s restart. Had the operator not looked, the unpushed commit would have been one more drain away from a hand recovery, which is what happened to rictus on ce-ymot. **What I could not determine from the log,** and I am not going to guess: why `poolDesired` dropped to 1. Nothing in the celilo store changed in that reconcile, and furiosa was inside its protection window, so neither the cache-lag path nor the protection-window expiry explains it on its face. `recordReconcileTraceInputs` (`cmd/gc/city_runtime.go:2586`) already captures `desiredState`, `poolDesired` and `workSet` per tick. Dumping that trace for one drain tick would settle both halves at once: what the count was built from, and which request the accepted slot named. Filed from gc-scratch, which continues to operate the celilo polecat pool at size one to two for this reason.

forgejo-admin commented

2026-09-09 02:11:05 +00:00

Author

Owner

Counter-measurement, 2026-09-09: five simultaneous releases into an empty pool drained nothing

Run deliberately at peba's request to test the "never batch" rule this issue produced. One trial, reported with its limits.

Method

Baseline at 01:37:35Z, chosen to reproduce the condition the earlier finding called worst — an empty pool:

live polecats           0
unheld pool demand      0
pool max                5

At 01:38:06Z, five celilo beads were released together (hold:mayor removed, gc.routed_to set to celilo/gastown.polecat): ce-tgz7, ce-vchn, ce-0f5p, ce-cq3i, ce-6t2s. Demand was confirmed with the supervisor's own exclusions (--unassigned --exclude-label hold:mayor --exclude-label hold:external), not a naive ready query, which ignores the hold labels and overcounts.

Result

All five seats came up:

nux      success                14.644s
slit     success                14.706s
capable  success                15.024s
rictus   success                15.374s
furiosa  start_error_converged   3.677s  -> retried, success 14.95s

First claim 01:39:56Z, all five holding claims by 01:47:51Z:

nux=ce-0f5p  slit=ce-cq3i  furiosa=ce-prr5  capable=ce-tgz7  rictus=ce-vchn

Zero drains cost a claim. Over the next 30 minutes the pool cycled normally: polecats finished, handed branches to the refinery, respawned, and took the next queued bead. Three beads went from release to pushed-and-in-the-merge-gate inside the window.

The one drain that did touch live work, and what triggered it

At ~01:41 rictus was drained mid-claim. The cause was mine: I routed a sixth bead (ce-prr5) at 01:40:50, about ten seconds earlier. I had assumed a sixth bead was harmless because the pool caps at 5 — wrong, because it is the spawn request that drains, not the seat count.

rictus respawned in 20.5s and kept ce-vchn, consistent with this issue's note that the assignee is the agent name so a respawn under the same name re-matches the claim.

A second drain, furiosa at ~02:09, held no claim at all — it had already handed off, idled, and drained. Benign. Counting all drains rather than drains that cost work will overstate this issue's impact.

What this suggests, and what it does not

Suggests: the trigger is adding demand while polecats are mid-work, not batch size at release. Releasing N into an idle pool behaved fine at N=5. If that holds, the one-at-a-time throttle described in this issue's earlier comments costs four of five seats to avoid a hazard that fires on a different action.

Does not establish: this is a single trial. It should be repeated before the operating rule changes.

A likely confounder in the earlier measurements, worth checking

The 2026-09-06 observations were taken on a city that may have been in a different state. Tonight, before this run, celilo's pool was spawning nothing at all because an unrelated rig with an uninitialised beads store made the demand read PARTIAL city-wide and buildDesiredState refuses fresh creates on a partial read (filed as #24). Spawn behaviour measured while that interlock was tripping intermittently would not be comparable. Anyone re-testing should first confirm ~/.gc/supervisor.log carries no demand read partial or fresh create blocked lines for the window.

## Counter-measurement, 2026-09-09: five simultaneous releases into an empty pool drained nothing Run deliberately at peba's request to test the "never batch" rule this issue produced. One trial, reported with its limits. ### Method Baseline at 01:37:35Z, chosen to reproduce the condition the earlier finding called worst — an **empty** pool: ``` live polecats 0 unheld pool demand 0 pool max 5 ``` At 01:38:06Z, five celilo beads were released together (`hold:mayor` removed, `gc.routed_to` set to `celilo/gastown.polecat`): `ce-tgz7`, `ce-vchn`, `ce-0f5p`, `ce-cq3i`, `ce-6t2s`. Demand was confirmed with the supervisor's own exclusions (`--unassigned --exclude-label hold:mayor --exclude-label hold:external`), not a naive ready query, which ignores the hold labels and overcounts. ### Result All five seats came up: ``` nux success 14.644s slit success 14.706s capable success 15.024s rictus success 15.374s furiosa start_error_converged 3.677s -> retried, success 14.95s ``` First claim 01:39:56Z, all five holding claims by 01:47:51Z: ``` nux=ce-0f5p slit=ce-cq3i furiosa=ce-prr5 capable=ce-tgz7 rictus=ce-vchn ``` **Zero drains cost a claim.** Over the next 30 minutes the pool cycled normally: polecats finished, handed branches to the refinery, respawned, and took the next queued bead. Three beads went from release to pushed-and-in-the-merge-gate inside the window. ### The one drain that did touch live work, and what triggered it At ~01:41 `rictus` was drained mid-claim. The cause was mine: I routed a sixth bead (`ce-prr5`) at 01:40:50, about ten seconds earlier. I had assumed a sixth bead was harmless because the pool caps at 5 — wrong, because it is the spawn *request* that drains, not the seat count. `rictus` respawned in 20.5s and **kept** `ce-vchn`, consistent with this issue's note that the assignee is the agent name so a respawn under the same name re-matches the claim. A second drain, `furiosa` at ~02:09, held no claim at all — it had already handed off, idled, and drained. Benign. Counting all drains rather than drains that cost work will overstate this issue's impact. ### What this suggests, and what it does not Suggests: the trigger is **adding demand while polecats are mid-work**, not batch size at release. Releasing N into an idle pool behaved fine at N=5. If that holds, the one-at-a-time throttle described in this issue's earlier comments costs four of five seats to avoid a hazard that fires on a different action. Does not establish: this is a single trial. It should be repeated before the operating rule changes. ### A likely confounder in the earlier measurements, worth checking The 2026-09-06 observations were taken on a city that may have been in a different state. Tonight, before this run, celilo's pool was spawning **nothing at all** because an unrelated rig with an uninitialised beads store made the demand read PARTIAL city-wide and `buildDesiredState` refuses fresh creates on a partial read (filed as #24). Spawn behaviour measured while that interlock was tripping intermittently would not be comparable. Anyone re-testing should first confirm `~/.gc/supervisor.log` carries no `demand read partial` or `fresh create blocked` lines for the window.

forgejo-admin referenced this issue

2026-09-09 02:58:05 +00:00

Pool triggers demand nine worktree-provenance keys that only gc worktree ensure writes, so a pack using worktree-setup.sh stalls its pool permanently #7

forgejo-admin commented

2026-09-11 20:48:24 +00:00

Author

Owner

Measured again on 2026-09-11: three drains of polecats holding claims, in 16 minutes. Two of them held uncommitted work. Times are UTC.

Context. The OpenRouter account ran out of credits at 18:58 and every pi agent stalled. Credits came back at 20:21, and the mayor typed a resume message into the refinery and four polecat panes at 20:21:12.

furiosa, a fresh spawn (gs-ouxuc7, transcript starts 20:22:46). It claimed ce-d5q5 at 20:23:40 and read the claim back at 20:23:48 (in_progress, assignee celilo/gastown.furiosa). supervisor.log then has Draining session 'celilo--gastown__furiosa': no-wake-reason, and ce-d5q5 was open and unassigned again at 20:24:48. About a minute from claim to release, the shape in this issue's title.
nux claimed ce-wlrd at 20:22:04 and worked it for seven minutes. Its last transcript entry is a file edit at 20:29:19. supervisor.log has Draining session 'celilo--gastown__nux': no-wake-reason, and ce-wlrd was open and unassigned at 20:29:31. Two new pool sessions (slit gs-mjbr3n, furiosa gs-nmjwdj) were live by about 20:31. nux left four modified files and two new ones uncommitted, and had pushed nothing.
slit (gs-mjbr3n) claimed ce-d5q5 at about 20:30. Its last transcript entry starts a typecheck at 20:37:15, right after an edit. supervisor.log has Draining session 'celilo--gastown__slit': no-wake-reason. The pool read one live polecat at 20:39:09, a new furiosa session's transcript starts at 20:39:18, and that session held ce-d5q5 by 20:39:50. slit left eight files uncommitted (four modified, four new).

No work was lost, and only because the next polecat chose to salvage it. One furiosa finished nux's work inside nux's worktree. The next copied slit's eight files into its own worktree, then ran git worktree remove --force on slit's worktree. That removal stopped partway and left 846 MB at slit's path, which worktree-setup.sh stages aside on the next spawn (the non-empty-directory branch at line 99).

Every surface read healthy throughout. The only records are the pi transcripts under ~/.pi/agent/sessions/ and ~/.gc/supervisor.log, which needs rg -a because the file holds NUL bytes.

(mayor, 2026-09-11)

Measured again on 2026-09-11: three drains of polecats holding claims, in 16 minutes. Two of them held uncommitted work. Times are UTC. Context. The OpenRouter account ran out of credits at 18:58 and every pi agent stalled. Credits came back at 20:21, and the mayor typed a resume message into the refinery and four polecat panes at 20:21:12. 1. furiosa, a fresh spawn (gs-ouxuc7, transcript starts 20:22:46). It claimed ce-d5q5 at 20:23:40 and read the claim back at 20:23:48 (in_progress, assignee celilo/gastown.furiosa). supervisor.log then has `Draining session 'celilo--gastown__furiosa': no-wake-reason`, and ce-d5q5 was open and unassigned again at 20:24:48. About a minute from claim to release, the shape in this issue's title. 2. nux claimed ce-wlrd at 20:22:04 and worked it for seven minutes. Its last transcript entry is a file edit at 20:29:19. supervisor.log has `Draining session 'celilo--gastown__nux': no-wake-reason`, and ce-wlrd was open and unassigned at 20:29:31. Two new pool sessions (slit gs-mjbr3n, furiosa gs-nmjwdj) were live by about 20:31. nux left four modified files and two new ones uncommitted, and had pushed nothing. 3. slit (gs-mjbr3n) claimed ce-d5q5 at about 20:30. Its last transcript entry starts a typecheck at 20:37:15, right after an edit. supervisor.log has `Draining session 'celilo--gastown__slit': no-wake-reason`. The pool read one live polecat at 20:39:09, a new furiosa session's transcript starts at 20:39:18, and that session held ce-d5q5 by 20:39:50. slit left eight files uncommitted (four modified, four new). No work was lost, and only because the next polecat chose to salvage it. One furiosa finished nux's work inside nux's worktree. The next copied slit's eight files into its own worktree, then ran `git worktree remove --force` on slit's worktree. That removal stopped partway and left 846 MB at slit's path, which worktree-setup.sh stages aside on the next spawn (the non-empty-directory branch at line 99). Every surface read healthy throughout. The only records are the pi transcripts under ~/.pi/agent/sessions/ and ~/.gc/supervisor.log, which needs `rg -a` because the file holds NUL bytes. (mayor, 2026-09-11)

forgejo-admin referenced this issue

2026-09-21 04:39:21 +00:00

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32

forgejo-admin referenced this issue

2026-09-21 04:54:28 +00:00

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32