celilo/gascity

Fork 0

A session can go deaf while every surface reads healthy: 16 nudges sat never-attempted for 3 hours, and `gc session wake` did not clear it #28

New issue

Open

opened 2026-09-10 02:16:48 +00:00 by forgejo-admin · 1 comment

forgejo-admin commented

2026-09-10 02:16:48 +00:00

Owner

What happened

The celilo refinery went deaf at 2026-09-09T23:14Z and stayed deaf for nearly
three hours. It is the city's serial merge bottleneck, so nothing merged in that
window and nobody could tell it to.

Every surface said it was fine:

gc session list        active
ps                     its `gc nudge poll` alive, pid 83737, 23h uptime
gc session peek        a complete, plausible, recent status report on screen
gc doctor              no complaint

The only statement of the truth was in .gc/nudges/state.json:

total pending: 16
all 16 for celilo/gastown.refinery
all 16 last_attempt_at = 0001-01-01T00:00:00Z   (NEVER ATTEMPTED)
oldest created_at     = 2026-09-09T23:14:55Z

Three of those sixteen were the mayor telling it that a P0 bead was blocking an
overnight run. They were never attempted, so gc session nudge reported success
sixteen times and delivered nothing.

The tell, for anyone debugging this next

A wedged session's token counters are frozen. A slow one's climb. Two peeks
thirty minutes apart returned byte-identical output including the counter line:

↑7.3M ↓106k R47M CH99.9% $1.283 18.2%/1.0M

Identical cost and identical token totals across thirty minutes is not a slow
agent. It is an agent that has not run.

The corroborating second instrument is the nudge backlog, because a healthy
session drains that queue. Frozen counters ALONE are not enough: a genuinely
idle session with nothing to do also shows frozen counters and is fine. It is
frozen counters PLUS an undrained backlog that means deaf.

`gc session wake` does not fix it. `gc session reset` does.

Measured in that order, with a baseline captured first:

baseline            ↑7.3M ↓106k R47M $1.283, 16 pending
gc session wake     "wake requested"
+100s               ↑7.3M ↓106k R47M $1.283, 16 pending   <- no change
gc session reset    "reset requested"
+200s               ↑255k ↓10k R1.8M $0.049, 0 pending    <- fresh, drained

So wake returns a success line and does nothing for this state, which makes it
a third surface that reads healthy while the session is not.

Note the reset DISCARDED the sixteen queued messages rather than delivering
them. That is arguably right for a fresh-mode session, but it means anything
important said to a wedged agent is lost, not deferred. Put it on the bead
instead, which is what finally worked here.

Why this is worse than it looks

gc session nudge to a non-claude provider already prints "live delivery is
unsupported for pi; the queued dispatcher delivers it". That line reads as a
reassurance. This city's own city.toml carries a comment asserting the queued
dispatcher does deliver, measured 2026-09-06. Both are true when the session is
healthy and both are false here, with no signal to distinguish the cases.

What would fix it

gc session wake should handle a session whose runtime is wedged, or say
plainly that it cannot and name reset as the remedy. Returning "wake
requested" for a no-op is the core of the problem.
Surface the backlog. gc session list or gc doctor showing a
pending-nudge count per session would have made this visible in one command.
A count of never-attempted nudges older than an hour is an unambiguous defect
signal.
Age out or escalate never-attempted nudges rather than letting them
accumulate silently to sixteen.

Workaround in use

assets/scripts/overnight-grind-watch.sh in the gc-scratch city compares the
refinery's counter line against the previous cycle and resets it when the
counters are frozen AND at least three nudges sit never-attempted. It
deliberately does not reset on frozen counters alone, because an idle session
looks the same and resetting a healthy agent costs a conversation.

Reported by the mayor, gc-scratch, 2026-09-10.

## What happened The celilo refinery went deaf at 2026-09-09T23:14Z and stayed deaf for nearly three hours. It is the city's serial merge bottleneck, so nothing merged in that window and nobody could tell it to. Every surface said it was fine: gc session list active ps its `gc nudge poll` alive, pid 83737, 23h uptime gc session peek a complete, plausible, recent status report on screen gc doctor no complaint The only statement of the truth was in `.gc/nudges/state.json`: total pending: 16 all 16 for celilo/gastown.refinery all 16 last_attempt_at = 0001-01-01T00:00:00Z (NEVER ATTEMPTED) oldest created_at = 2026-09-09T23:14:55Z Three of those sixteen were the mayor telling it that a P0 bead was blocking an overnight run. They were never attempted, so `gc session nudge` reported success sixteen times and delivered nothing. ## The tell, for anyone debugging this next **A wedged session's token counters are frozen. A slow one's climb.** Two peeks thirty minutes apart returned byte-identical output including the counter line: ↑7.3M ↓106k R47M CH99.9% $1.283 18.2%/1.0M Identical cost and identical token totals across thirty minutes is not a slow agent. It is an agent that has not run. The corroborating second instrument is the nudge backlog, because a healthy session drains that queue. Frozen counters ALONE are not enough: a genuinely idle session with nothing to do also shows frozen counters and is fine. It is frozen counters PLUS an undrained backlog that means deaf. ## `gc session wake` does not fix it. `gc session reset` does. Measured in that order, with a baseline captured first: baseline ↑7.3M ↓106k R47M $1.283, 16 pending gc session wake "wake requested" +100s ↑7.3M ↓106k R47M $1.283, 16 pending <- no change gc session reset "reset requested" +200s ↑255k ↓10k R1.8M $0.049, 0 pending <- fresh, drained So `wake` returns a success line and does nothing for this state, which makes it a third surface that reads healthy while the session is not. Note the reset DISCARDED the sixteen queued messages rather than delivering them. That is arguably right for a fresh-mode session, but it means anything important said to a wedged agent is lost, not deferred. Put it on the bead instead, which is what finally worked here. ## Why this is worse than it looks `gc session nudge` to a non-claude provider already prints "live delivery is unsupported for pi; the queued dispatcher delivers it". That line reads as a reassurance. This city's own `city.toml` carries a comment asserting the queued dispatcher does deliver, measured 2026-09-06. Both are true when the session is healthy and both are false here, with no signal to distinguish the cases. ## What would fix it 1. **`gc session wake` should handle a session whose runtime is wedged**, or say plainly that it cannot and name `reset` as the remedy. Returning "wake requested" for a no-op is the core of the problem. 2. **Surface the backlog.** `gc session list` or `gc doctor` showing a pending-nudge count per session would have made this visible in one command. A count of never-attempted nudges older than an hour is an unambiguous defect signal. 3. **Age out or escalate never-attempted nudges** rather than letting them accumulate silently to sixteen. ## Workaround in use `assets/scripts/overnight-grind-watch.sh` in the gc-scratch city compares the refinery's counter line against the previous cycle and resets it when the counters are frozen AND at least three nudges sit never-attempted. It deliberately does not reset on frozen counters alone, because an idle session looks the same and resetting a healthy agent costs a conversation. Reported by the mayor, gc-scratch, 2026-09-10.

forgejo-admin commented

2026-09-10 15:03:44 +00:00

Author

Owner

Tonight's instance, and a workaround that reaches the agent

At 14:59Z .gc/nudges/state.json held eight queued nudges, created between 06:15Z and 14:57Z. Every one had last_attempt_at and claimed_at at the zero time. Several target sessions that are asleep or gone, where waiting is expected. Two targeted the live celilo refinery (gs-wisp-45y3e77, continuation epoch 39):

07:30:51Z  from a polecat: ce-esbz ready for merge, branch polecat/ce-esbz at 8f9322a5
14:57:23Z  from the mayor: resume the queue

The refinery's poller ran the whole time (gc nudge poll --city /Users/pbanka/gc-scratch --session celilo--gastown__refinery gs-wisp-45y3e77, up 7.5 hours). The refinery itself had been idle at its pi prompt since 08:58Z, when its model calls to OpenRouter failed (two timeouts, a connection error, then "Retry failed after 3 attempts"). herdr reported the pane agent_status: done. So the session was as idle as a session gets, and the poller still never attempted either nudge. tryDeliverQueuedNudgesByPoller (cmd/gc/cmd_nudge.go:1413) returns before it claims anything unless the live generation matches and pollerSessionIdleEnough passes, so one of those two checks is failing for this session. I did not dig further.

The cost: the refinery sat on a queue of nine for six hours. That queue held the P1 fix for 19 of the 23 failures in last night's e2e grind. gc session nudge printed Queued nudge ... (live delivery is unsupported for pi; the queued dispatcher delivers it), which reads like delivery and was not.

A workaround that works

Type into the agent's herdr pane directly:

herdr pane list                         # find the pane whose cwd is the agent's worktree
herdr pane read <pane_id>               # confirm it is the agent you mean
herdr pane run <pane_id> '<message>'    # sends the text and Enter

At 15:02Z the refinery's pane went to agent_status: working, and its transcript recorded the message as a user turn within seconds. Recorded in gc-scratch CLAUDE.md under Tool traps.

Reported by the mayor, gc-scratch, 2026-09-10.

## Tonight's instance, and a workaround that reaches the agent At 14:59Z `.gc/nudges/state.json` held eight queued nudges, created between 06:15Z and 14:57Z. Every one had `last_attempt_at` and `claimed_at` at the zero time. Several target sessions that are asleep or gone, where waiting is expected. Two targeted the live celilo refinery (`gs-wisp-45y3e77`, continuation epoch 39): 07:30:51Z from a polecat: ce-esbz ready for merge, branch polecat/ce-esbz at 8f9322a5 14:57:23Z from the mayor: resume the queue The refinery's poller ran the whole time (`gc nudge poll --city /Users/pbanka/gc-scratch --session celilo--gastown__refinery gs-wisp-45y3e77`, up 7.5 hours). The refinery itself had been idle at its pi prompt since 08:58Z, when its model calls to OpenRouter failed (two timeouts, a connection error, then "Retry failed after 3 attempts"). herdr reported the pane `agent_status: done`. So the session was as idle as a session gets, and the poller still never attempted either nudge. `tryDeliverQueuedNudgesByPoller` (`cmd/gc/cmd_nudge.go:1413`) returns before it claims anything unless the live generation matches and `pollerSessionIdleEnough` passes, so one of those two checks is failing for this session. I did not dig further. The cost: the refinery sat on a queue of nine for six hours. That queue held the P1 fix for 19 of the 23 failures in last night's e2e grind. `gc session nudge` printed `Queued nudge ... (live delivery is unsupported for pi; the queued dispatcher delivers it)`, which reads like delivery and was not. ## A workaround that works Type into the agent's herdr pane directly: herdr pane list # find the pane whose cwd is the agent's worktree herdr pane read <pane_id> # confirm it is the agent you mean herdr pane run <pane_id> '<message>' # sends the text and Enter At 15:02Z the refinery's pane went to `agent_status: working`, and its transcript recorded the message as a user turn within seconds. Recorded in gc-scratch CLAUDE.md under Tool traps. Reported by the mayor, gc-scratch, 2026-09-10.

forgejo-admin referenced this issue

2026-09-21 04:37:52 +00:00

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#28

No description provided.

Rows
Columns

A session can go deaf while every surface reads healthy: 16 nudges sat never-attempted for 3 hours, and gc session wake did not clear it #28