celilo/gascity

Fork 0

An always-on session is cycled for "bead reassign" when its own bead CLOSES, so finishing work destroys the conversation 34 seconds later #26

New issue

Open

opened 2026-09-09 21:29:07 +00:00 by forgejo-admin · 1 comment

forgejo-admin commented

2026-09-09 21:29:07 +00:00

Owner

What happens

The supervisor kills and restarts an always-on named session whenever the
identity of "its bead" changes, and it counts CLOSED beads when it decides
that. So an agent closing its own work destroys its own conversation about
thirty seconds later.

Measured on the mayor, 2026-09-09 (all times UTC):

21:13:07  the mayor closes ce-mcpp (its own bead, work finished)
21:13:41  supervisor.log: Cycled fresh-mode session 'gastown__mayor'
          for bead reassign: ce-xki4 -> ce-mcpp
21:14:31  a brand new mayor session starts, remembering nothing
          (session lifecycle: op=start outcome=success duration=27.328s)

Both beads named in that line are closed. ce-xki4 closed 2026-09-07T17:37:10Z.
ce-mcpp closed 2026-09-09T21:13:07Z, which is the event that triggered the
cycle. Nothing was assigned to the mayor. The set the supervisor reads its
"current bead" from includes closed beads, so the top row changed when ce-mcpp
got its close timestamp, and that alone was read as a reassignment.

Because the session is wake_mode = "fresh", "cycle" means throw the
conversation away and start over.

It is not just the mayor

Nine of these in the log, across every always-on role:

2026/09/07 04:09  gastown__mayor              ce-jom4 -> ce-xki4
2026/09/07 13:43  celilo--gastown__refinery   ce-wisp-zn6 -> ce-wisp-r7l
2026/09/07 15:15  gastown__deacon             gs-wisp-6qb0wj0 -> gs-wisp-578pd
2026/09/07 18:23  celilo--gastown__witness    ce-wisp-19o -> ce-wisp-2h2
2026/09/07 18:30  celilo--gastown__witness    ce-wisp-2h2 -> ce-wisp-yxl
2026/09/07 18:32  gastown__deacon             gs-wisp-578pd -> gs-wisp-z5vr4
2026/09/07 19:31  celilo--gastown__refinery   ce-wisp-r7l -> ce-wisp-46d
2026/09/07 19:37  celilo--gastown__refinery   ce-wisp-46d -> ce-wisp-n40
2026/09/09 21:13  gastown__mayor              ce-xki4 -> ce-mcpp

The witness and refinery ones are on *-wisp-* beads. Those are ephemeral
tracking beads and they rotate as a matter of course, so the refinery loses its
conversation on a schedule set by wisp turnover. The refinery is the one role
in the city that most needs to remember what it just gated, and it is being
restarted three times in six hours by bookkeeping.

Why the config cannot absorb it

The pinned gastown pack sets wake_mode = "fresh" on mayor, deacon, witness,
refinery, boot, dog and polecat. mode = "always" plus wake_mode = "fresh" is
the exact combination gc's own linter warns about, and it prints that warning on
every supervisor tick:

gc supervisor: warning: named_session "gastown.mayor": mode "always" with
wake_mode "fresh" on template "gastown.mayor" starts a fresh provider
session after every drain; use only for a deliberate restart-per-cycle actor

An operator cannot fix this from city.toml. wake_mode is not patchable through
[[rigs.patches]] (the schema carries no wake_mode field). The only local
workaround is a city-local agents/<name>/agent.toml that overrides it, which is
what the crew agents in this city already do (wake_mode = "resume"), and that
does not reach the pack roles.

Two separate fixes, and the first one is the real one

Do not cycle on a closed bead. Whatever query feeds the reassign check
should exclude status=closed. A bead the agent just finished is not a new
assignment. This alone removes the mayor and deacon cases.
Do not cycle on a wisp. An ephemeral tracking bead rotating is not a
reassignment either. Excluding ephemeral=true removes the witness and
refinery cases.

Separately, the pack shipping always + fresh on four long-lived coordinator
roles is worth a look, since it is the amplifier: without it, a spurious reassign
would cost a resume rather than the whole conversation. That is a
gascity-packs change and it needs a deliberate re-pin, so it is not the fast fix.

Cost

Every cycle is a cold start. The new session re-reads STANDING-DECISIONS.md,
CLAUDE.md, its inbox and its queue before it can do anything, on a host that is
frequently the bottleneck for every merge in the city. It also silently discards
whatever the killed session was mid-way through, with no record that it did.

Reported by the mayor, gc-scratch, 2026-09-09.

## What happens The supervisor kills and restarts an always-on named session whenever the identity of "its bead" changes, and it counts CLOSED beads when it decides that. So an agent closing its own work destroys its own conversation about thirty seconds later. Measured on the mayor, 2026-09-09 (all times UTC): 21:13:07 the mayor closes ce-mcpp (its own bead, work finished) 21:13:41 supervisor.log: Cycled fresh-mode session 'gastown__mayor' for bead reassign: ce-xki4 -> ce-mcpp 21:14:31 a brand new mayor session starts, remembering nothing (session lifecycle: op=start outcome=success duration=27.328s) Both beads named in that line are closed. ce-xki4 closed 2026-09-07T17:37:10Z. ce-mcpp closed 2026-09-09T21:13:07Z, which is the event that triggered the cycle. Nothing was assigned to the mayor. The set the supervisor reads its "current bead" from includes closed beads, so the top row changed when ce-mcpp got its close timestamp, and that alone was read as a reassignment. Because the session is `wake_mode = "fresh"`, "cycle" means throw the conversation away and start over. ## It is not just the mayor Nine of these in the log, across every always-on role: 2026/09/07 04:09 gastown__mayor ce-jom4 -> ce-xki4 2026/09/07 13:43 celilo--gastown__refinery ce-wisp-zn6 -> ce-wisp-r7l 2026/09/07 15:15 gastown__deacon gs-wisp-6qb0wj0 -> gs-wisp-578pd 2026/09/07 18:23 celilo--gastown__witness ce-wisp-19o -> ce-wisp-2h2 2026/09/07 18:30 celilo--gastown__witness ce-wisp-2h2 -> ce-wisp-yxl 2026/09/07 18:32 gastown__deacon gs-wisp-578pd -> gs-wisp-z5vr4 2026/09/07 19:31 celilo--gastown__refinery ce-wisp-r7l -> ce-wisp-46d 2026/09/07 19:37 celilo--gastown__refinery ce-wisp-46d -> ce-wisp-n40 2026/09/09 21:13 gastown__mayor ce-xki4 -> ce-mcpp The witness and refinery ones are on `*-wisp-*` beads. Those are ephemeral tracking beads and they rotate as a matter of course, so the refinery loses its conversation on a schedule set by wisp turnover. The refinery is the one role in the city that most needs to remember what it just gated, and it is being restarted three times in six hours by bookkeeping. ## Why the config cannot absorb it The pinned gastown pack sets `wake_mode = "fresh"` on mayor, deacon, witness, refinery, boot, dog and polecat. `mode = "always"` plus `wake_mode = "fresh"` is the exact combination gc's own linter warns about, and it prints that warning on every supervisor tick: gc supervisor: warning: named_session "gastown.mayor": mode "always" with wake_mode "fresh" on template "gastown.mayor" starts a fresh provider session after every drain; use only for a deliberate restart-per-cycle actor An operator cannot fix this from city.toml. `wake_mode` is not patchable through `[[rigs.patches]]` (the schema carries no `wake_mode` field). The only local workaround is a city-local `agents/<name>/agent.toml` that overrides it, which is what the crew agents in this city already do (`wake_mode = "resume"`), and that does not reach the pack roles. ## Two separate fixes, and the first one is the real one 1. **Do not cycle on a closed bead.** Whatever query feeds the reassign check should exclude `status=closed`. A bead the agent just finished is not a new assignment. This alone removes the mayor and deacon cases. 2. **Do not cycle on a wisp.** An ephemeral tracking bead rotating is not a reassignment either. Excluding `ephemeral=true` removes the witness and refinery cases. Separately, the pack shipping `always` + `fresh` on four long-lived coordinator roles is worth a look, since it is the amplifier: without it, a spurious reassign would cost a resume rather than the whole conversation. That is a gascity-packs change and it needs a deliberate re-pin, so it is not the fast fix. ## Cost Every cycle is a cold start. The new session re-reads STANDING-DECISIONS.md, CLAUDE.md, its inbox and its queue before it can do anything, on a host that is frequently the bottleneck for every merge in the city. It also silently discards whatever the killed session was mid-way through, with no record that it did. Reported by the mayor, gc-scratch, 2026-09-09.

forgejo-admin commented

2026-09-10 11:16:13 +00:00

Author

Owner

A second instance, and the trigger is the assignment, not the close

2026-09-10, the mayor again. No bead closed this time.

2026-09-09 21:13:07Z  ce-mcpp closes. The mayor's currently_processing_bead_id keeps saying ce-mcpp.
2026-09-10 07:42:50Z  the mayor parks ce-hct1 on itself for peba's review:
                      gc bd update ce-hct1 --assignee gastown.mayor --add-label hold:mayor ...
07:44:56Z             the beads cache picks up the write (bead.updated on ce-hct1)
07:45:16Z             Cycled fresh-mode session 'gastown__mayor' for bead reassign: ce-mcpp → ce-hct1
07:45:21Z             session bead gs-yxywi: state=asleep, sleep_reason=runtime-missing
07:45:35Z             session.woke, a new conversation that remembers nothing

ce-hct1 was open the whole time. The only thing that changed was its assignee. The mayor died while it was answering a question from peba.

The 09-09 instance was an assignment too

The body says nothing was assigned to the mayor on 09-09. The mayor's own transcript says it assigned the bead to itself fifteen seconds before closing it:

21:12:52Z  gc bd --rig celilo update ce-mcpp --assignee "gastown.mayor" --set-metadata "gc.work_outcome=no-op"
           gc bd --rig celilo close ce-mcpp --reason ...

The session's record at that moment was ce-xki4, closed on 09-07. The assignment made ce-mcpp the anchor. ce-mcpp was not ce-xki4, and that is the reassign the log line names. The close came afterwards and did not change the anchor.

What is wrong

cmd/gc/compute_awake_set.go:406 sets RequiresFreshCycle whenever the anchor differs from currently_processing_bead_id, and session_reconciler.go:3736 then kills the process when the session is wake_mode = "fresh". The record is not cleared when its bead closes. ce-mcpp was still the mayor's record at 07:45:16Z, ten and a half hours after it closed. (Its only writers in the source are the wake path, recordCurrentBeadIDOnWake, and the cycle itself, session_bead_cycle.go:104.)

An always-on session that usually holds no assigned bead, like the mayor, therefore carries a stale record forever. The first bead anyone assigns to it afterwards reads as a reassignment, and cycleAliveSessionForFreshReassign kills the live process in the middle of a turn. It does not drain and does not wait for an ack.

So fix 1 in the body (exclude closed beads from the anchor query) would not have stopped this one, because ce-hct1 was open. Two things would have: do not cycle when the recorded bead is closed (the session finished that bead, so a new one is not a reassignment), or never kill a session mid-turn for a reassign.

The hold:mayor label, added in the same command, did not stop it.

The config can absorb it after all

The body says wake_mode is not patchable. That is true of [[rigs.patches]]. It is not true of [[patches.agent]], which carries wake_mode (internal/config/patch.go:139, applied at :623).

Proven on a staged copy of this city, loaded with the supervisor's own binary (/Users/pbanka/.local/bin/gc, a link to /Users/pbanka/go/bin/gc). Adding wake_mode = "resume" to the existing [[patches.agent]] name = "mayor" block loads with exit 0. The merged config changes in exactly two places: the mayor's wake_mode goes from fresh to resume, and the mayor's always-plus-fresh lint warning disappears. Since the reconciler only cycles fresh sessions, a resume mayor would survive an assignment. Not applied to the live city. That is peba's call.

Reported by the mayor, gc-scratch, 2026-09-10.

## A second instance, and the trigger is the assignment, not the close 2026-09-10, the mayor again. No bead closed this time. 2026-09-09 21:13:07Z ce-mcpp closes. The mayor's currently_processing_bead_id keeps saying ce-mcpp. 2026-09-10 07:42:50Z the mayor parks ce-hct1 on itself for peba's review: gc bd update ce-hct1 --assignee gastown.mayor --add-label hold:mayor ... 07:44:56Z the beads cache picks up the write (bead.updated on ce-hct1) 07:45:16Z Cycled fresh-mode session 'gastown__mayor' for bead reassign: ce-mcpp → ce-hct1 07:45:21Z session bead gs-yxywi: state=asleep, sleep_reason=runtime-missing 07:45:35Z session.woke, a new conversation that remembers nothing ce-hct1 was open the whole time. The only thing that changed was its assignee. The mayor died while it was answering a question from peba. ## The 09-09 instance was an assignment too The body says nothing was assigned to the mayor on 09-09. The mayor's own transcript says it assigned the bead to itself fifteen seconds before closing it: 21:12:52Z gc bd --rig celilo update ce-mcpp --assignee "gastown.mayor" --set-metadata "gc.work_outcome=no-op" gc bd --rig celilo close ce-mcpp --reason ... The session's record at that moment was ce-xki4, closed on 09-07. The assignment made ce-mcpp the anchor. ce-mcpp was not ce-xki4, and that is the reassign the log line names. The close came afterwards and did not change the anchor. ## What is wrong `cmd/gc/compute_awake_set.go:406` sets `RequiresFreshCycle` whenever the anchor differs from `currently_processing_bead_id`, and `session_reconciler.go:3736` then kills the process when the session is `wake_mode = "fresh"`. The record is not cleared when its bead closes. ce-mcpp was still the mayor's record at 07:45:16Z, ten and a half hours after it closed. (Its only writers in the source are the wake path, `recordCurrentBeadIDOnWake`, and the cycle itself, `session_bead_cycle.go:104`.) An always-on session that usually holds no assigned bead, like the mayor, therefore carries a stale record forever. The first bead anyone assigns to it afterwards reads as a reassignment, and `cycleAliveSessionForFreshReassign` kills the live process in the middle of a turn. It does not drain and does not wait for an ack. So fix 1 in the body (exclude closed beads from the anchor query) would not have stopped this one, because ce-hct1 was open. Two things would have: do not cycle when the recorded bead is closed (the session finished that bead, so a new one is not a reassignment), or never kill a session mid-turn for a reassign. The `hold:mayor` label, added in the same command, did not stop it. ## The config can absorb it after all The body says `wake_mode` is not patchable. That is true of `[[rigs.patches]]`. It is not true of `[[patches.agent]]`, which carries `wake_mode` (`internal/config/patch.go:139`, applied at `:623`). Proven on a staged copy of this city, loaded with the supervisor's own binary (`/Users/pbanka/.local/bin/gc`, a link to `/Users/pbanka/go/bin/gc`). Adding `wake_mode = "resume"` to the existing `[[patches.agent]] name = "mayor"` block loads with exit 0. The merged config changes in exactly two places: the mayor's `wake_mode` goes from `fresh` to `resume`, and the mayor's always-plus-fresh lint warning disappears. Since the reconciler only cycles `fresh` sessions, a resume mayor would survive an assignment. Not applied to the live city. That is peba's call. Reported by the mayor, gc-scratch, 2026-09-10.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#26

No description provided.

Rows
Columns