Assigning a bead to an idle on_demand named session does not wake it, so a refinery sleeps on a full merge queue until a human nudges it #10

New issue

Open

opened 2026-09-03 04:59:38 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-03 04:59:38 +00:00

Owner

What happens

An on_demand named session ends a turn with "IDLE: no work, exiting turn". Work is then correctly assigned to it. Nothing wakes it. It sleeps on a full queue indefinitely, and every operator surface reads healthy.

This is not #9. In #9 the bead carried gc.routed_to, which a named session's hook query does not match, so the bead was never claimable. Here the beads are assignee = celilo/gastown.refinery, which is exactly the workaround #9 recommends, and the refinery's own find-work query matches them. The query is right. Nothing runs it.

Measured, 2026-09-02

Timeline on the gc-scratch city, celilo rig:

15:17 PDT  celilo/gastown.refinery ends its turn: "IDLE: no work, exiting turn"
           its own reasoning, from gc session peek: "the refinery is an
           on_demand named session, so the controller wakes me when work is
           assigned rather than a poured wisp doing it"
18:26 PDT  crew assigns ce-p98w (branch ce-fleetkey) to the refinery
     ...   three more assigned: ce-v7kc, ce-af4w, ce-0cdb
21:55 PDT  gc status: celilo/gastown.refinery running
           four beads, status open, assignee celilo/gastown.refinery:
             ce-v7kc  P1  branch=polecat/ce-v7kc
             ce-p98w  P1  branch=ce-fleetkey
             ce-af4w  P1  branch=ce-orchprim2
             ce-0cdb  P2  branch=polecat/ce-0cdb
           gc bd list --status in_progress --rig celilo -> empty
           the refinery is still parked at its input box
21:58 PDT  gc session nudge celilo/gastown.refinery "<the queue>"
           it wakes immediately and begins gating

Six and a half hours, four beads, one of them (ce-p98w) carrying a fix that unblocks an e2e suite that had been dead at stage 2.

The refinery's belief is in its own transcript and is the crux: it declined to pour a fresh patrol wisp because it expected the controller to wake it on assignment. That expectation is either wrong, or the wake exists and did not fire. Either way the two halves disagree and neither says so.

Why it is invisible

gc status shows celilo/gastown.refinery running. It is running. It is parked.
gc bd list --status in_progress returns empty, which reads as "queue drained" and is indistinguishable from "queue never started".
The beads look perfectly healthy: correct assignee, correct branch metadata, correct status.
Nothing logs a refusal, because nothing refused. No decision was made at all.

No surface says "this session has assigned work and is asleep". That is the gap.

Blast radius

Any on_demand named session, which on this city is the refineries and crew. The refinery is the worst case: under the no-pull-requests rule it is the only path to main, so a sleeping refinery halts all landing on that rig while reporting healthy.

It compounds with two issues filed today. #9 means routing does not reach a named session, so assignment is the only channel. This means assignment does not wake one. Between them, the only reliable way to get work to a named session is a human noticing and nudging.

Suggested fix

Wake on assignment. If the controller already intends to, and the refinery believes it does, find why it did not fire. If it does not, that is the feature.
Failing that, make the state visible. gc status should mark a session holding assigned, unstarted work while not running a turn, and gc doctor should check for it. The information is already in the bead store.
Consider a floor. An on_demand session with a non-empty queue could be woken on the same tick that evaluates pool demand. The scale check already runs every tick and already reads the bead store.

#9, routing does not reach a named session's hook. Same family, opposite half: that is "the bead is not claimable", this is "the bead is claimable and nobody looks".
#7, pool triggers stall on stale worktree evidence. Third variant of one shape today: work is dispatched, the write receipt is honest, the target never sees it.
ce-uh79 in the celilo bead store: the refinery's find-work query filters --status=open, so a bead that reaches in_progress on its hook and stops is invisible to it too. That is a pack defect and probably belongs in gascity-packs.

Environment

gc 88f3894ad-dirty, built 2026-08-31. Pack pinned at gastownhall/gascity-packs sha:3b3b89f2011e06d84459aa7bea1552382f13930a. macOS, herdr runtime. City /Users/pbanka/gc-scratch. Host load 6.6 on 10 cores at the time, so this is not starvation.

## What happens An `on_demand` named session ends a turn with "IDLE: no work, exiting turn". Work is then correctly assigned to it. Nothing wakes it. It sleeps on a full queue indefinitely, and every operator surface reads healthy. This is **not** #9. In #9 the bead carried `gc.routed_to`, which a named session's hook query does not match, so the bead was never claimable. Here the beads are `assignee = celilo/gastown.refinery`, which is exactly the workaround #9 recommends, and the refinery's own find-work query matches them. The query is right. Nothing runs it. ## Measured, 2026-09-02 Timeline on the gc-scratch city, celilo rig: ``` 15:17 PDT celilo/gastown.refinery ends its turn: "IDLE: no work, exiting turn" its own reasoning, from gc session peek: "the refinery is an on_demand named session, so the controller wakes me when work is assigned rather than a poured wisp doing it" 18:26 PDT crew assigns ce-p98w (branch ce-fleetkey) to the refinery ... three more assigned: ce-v7kc, ce-af4w, ce-0cdb 21:55 PDT gc status: celilo/gastown.refinery running four beads, status open, assignee celilo/gastown.refinery: ce-v7kc P1 branch=polecat/ce-v7kc ce-p98w P1 branch=ce-fleetkey ce-af4w P1 branch=ce-orchprim2 ce-0cdb P2 branch=polecat/ce-0cdb gc bd list --status in_progress --rig celilo -> empty the refinery is still parked at its input box 21:58 PDT gc session nudge celilo/gastown.refinery "<the queue>" it wakes immediately and begins gating ``` Six and a half hours, four beads, one of them (`ce-p98w`) carrying a fix that unblocks an e2e suite that had been dead at stage 2. The refinery's belief is in its own transcript and is the crux: it declined to pour a fresh patrol wisp *because* it expected the controller to wake it on assignment. That expectation is either wrong, or the wake exists and did not fire. Either way the two halves disagree and neither says so. ## Why it is invisible - `gc status` shows `celilo/gastown.refinery running`. It is running. It is parked. - `gc bd list --status in_progress` returns empty, which reads as "queue drained" and is indistinguishable from "queue never started". - The beads look perfectly healthy: correct assignee, correct `branch` metadata, correct status. - Nothing logs a refusal, because nothing refused. No decision was made at all. No surface says "this session has assigned work and is asleep". That is the gap. ## Blast radius Any `on_demand` named session, which on this city is the refineries and `crew`. The refinery is the worst case: under the no-pull-requests rule it is the only path to `main`, so a sleeping refinery halts all landing on that rig while reporting healthy. It compounds with two issues filed today. #9 means routing does not reach a named session, so assignment is the only channel. This means assignment does not wake one. Between them, the only reliable way to get work to a named session is a human noticing and nudging. ## Suggested fix 1. **Wake on assignment.** If the controller already intends to, and the refinery believes it does, find why it did not fire. If it does not, that is the feature. 2. **Failing that, make the state visible.** `gc status` should mark a session holding assigned, unstarted work while not running a turn, and `gc doctor` should check for it. The information is already in the bead store. 3. **Consider a floor.** An `on_demand` session with a non-empty queue could be woken on the same tick that evaluates pool demand. The scale check already runs every tick and already reads the bead store. ## Related - #9, routing does not reach a named session's hook. Same family, opposite half: that is "the bead is not claimable", this is "the bead is claimable and nobody looks". - #7, pool triggers stall on stale worktree evidence. Third variant of one shape today: work is dispatched, the write receipt is honest, the target never sees it. - `ce-uh79` in the celilo bead store: the refinery's find-work query filters `--status=open`, so a bead that reaches `in_progress` on its hook and stops is invisible to it too. That is a pack defect and probably belongs in `gascity-packs`. ## Environment gc `88f3894ad-dirty`, built 2026-08-31. Pack pinned at gastownhall/gascity-packs `sha:3b3b89f2011e06d84459aa7bea1552382f13930a`. macOS, herdr runtime. City `/Users/pbanka/gc-scratch`. Host load 6.6 on 10 cores at the time, so this is not starvation.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#10

No description provided.

Rows
Columns