celilo/gascity

Fork 0

A city.toml that fails to load still drains every running session for config drift, while the supervisor logs "keeping old config" #29

New issue

Open

opened 2026-09-10 07:23:56 +00:00 by forgejo-admin · 1 comment

forgejo-admin commented

2026-09-10 07:23:56 +00:00

Owner

What happened

At 2026-09-10 06:58:20Z the mayor added a crew member. It wrote a [[patches.agent]] block for crew-burner into city.toml before agents/crew-burner/agent.toml existed. That config cannot load, and the supervisor said so:

gc supervisor: config reload: parsing city.toml: applying patches: patches.agent[5]: agent "crew-burner" not found in merged config (keeping old config)

"Keeping old config" should mean nothing changes. Instead, four to nine seconds later, the supervisor drained seven running sessions:

23:58:24.1 PDT  gs-yxywi         gastown.mayor               config drift detected
23:58:24.8      gs-sq4mn         gastown.boot                config drift detected
23:58:25.3      gs-tmwm3         gastown.deacon              config drift detected
23:58:25.8      gs-mw0zkq        lunacycle/gastown.refinery  config drift detected
23:58:26.7      gs-2lv5io        crew-cli                    config drift detected
23:58:28.0      gs-wisp-45y3e77  celilo/gastown.refinery     config drift detected
23:58:28.9      gs-mwfn1         celilo/gastown.witness      config drift detected

The supervisor log shows what each one drifted to:

config-drift gastown__mayor: stored=v5:f874fc41fc current=v5:26a75e3704 cmd=""
config-drift gastown__boot: stored=v5:b82b75d735 current=v5:26a75e3704 cmd=""
config-drift gastown__deacon: stored=v5:6068bc2eef current=v5:26a75e3704 cmd=""
config-drift lunacycle--gastown__refinery: stored=v5:e5be5c0fee current=v5:26a75e3704 cmd=""
config-drift crew-cli: stored=v5:dd1cb012ed current=v5:26a75e3704 cmd=""
config-drift celilo--gastown__refinery: stored=v5:a46045ff1e current=v5:26a75e3704 cmd=""
config-drift celilo--gastown__witness: stored=v5:22da8c6614 current=v5:26a75e3704 cmd=""
  config-drift-diag: drifted fields: Command, CopyFiles, Env, FPExtra, OverlayProviders (and PreStart for both refineries and crew-cli)

Seven different agents cannot share one fingerprint. v5:26a75e3704 with an empty command is what a session hashes to when it is built from nothing. So the drift check built "current" from the config that had just failed to load, while the reload path kept the old config in memory. The mayor that made the edit was mid-conversation. Its transcript stops at 06:58:23Z, the drain was logged at 06:58:24Z, and the supervisor relaunched the mayor (session.woke 06:58:30Z, new process 06:58:49Z).

It happened before, the same way

2026-09-08 17:55 PDT. Three failed reloads read patches.agent[4]: agent "crew-bna" not found in merged config (keeping old config). Six sessions drained with current=v5:26a75e3704 cmd="" and were relaunched at 17:55:28 (supervisor.log lines 125256 to 125506 on this host).

The damage outlives the bad config

The relaunched sessions store the empty fingerprint as their starting config. After tonight's relaunch, gs-yxywi (mayor) and gs-tmwm3 (deacon) both carry:

started_config_hash = v5:26a75e3704c256abbb0719e6274cd69ab5953792c0d08d1ecf4eda085849bc34

So when the config loads again, every relaunched session drifts a second time, from the empty hash back to its real one. On 2026-09-08 at 18:00 that second round resolved as session_live re-applied for all six, without a restart.

And one bad patch fails every bd call

While city.toml cannot load, every gc and bd command fails with the same error, plain bead reads included:

gc session list: listing sessions: listing session beads by type: bd list: load hosted Beads credential configuration: applying patches: patches.agent[5]: agent "crew-burner" not found in merged config

The supervisor's order dispatch and its session builds fail the same way on every tick. So the agents it just relaunched cannot read or claim work until someone fixes the file. Reading the bead store should not depend on whether an agent patch resolves.

Expected

A reload that fails and keeps the old config should leave running sessions alone. The drift check should compare against the config the supervisor is actually running, or skip while the new config fails to load.

Workaround in this city

Never let the supervisor see a city.toml nobody has loaded. Stage the change in a copy of the city and run gc config show --city <copy> first. On 2026-09-10 that check failed on this exact error, and it passed on both the pre-add backup and the fix. And write agents/<name>/agent.toml before any city.toml block that names it. Recorded in gc-scratch CLAUDE.md under "Adding a crew member (mayor)".

Supervisor: /Users/pbanka/.local/bin/gc supervisor run. Session provider: herdr 0.8.2.

## What happened At 2026-09-10 06:58:20Z the mayor added a crew member. It wrote a `[[patches.agent]]` block for `crew-burner` into city.toml before `agents/crew-burner/agent.toml` existed. That config cannot load, and the supervisor said so: ``` gc supervisor: config reload: parsing city.toml: applying patches: patches.agent[5]: agent "crew-burner" not found in merged config (keeping old config) ``` "Keeping old config" should mean nothing changes. Instead, four to nine seconds later, the supervisor drained seven running sessions: ``` 23:58:24.1 PDT gs-yxywi gastown.mayor config drift detected 23:58:24.8 gs-sq4mn gastown.boot config drift detected 23:58:25.3 gs-tmwm3 gastown.deacon config drift detected 23:58:25.8 gs-mw0zkq lunacycle/gastown.refinery config drift detected 23:58:26.7 gs-2lv5io crew-cli config drift detected 23:58:28.0 gs-wisp-45y3e77 celilo/gastown.refinery config drift detected 23:58:28.9 gs-mwfn1 celilo/gastown.witness config drift detected ``` The supervisor log shows what each one drifted to: ``` config-drift gastown__mayor: stored=v5:f874fc41fc current=v5:26a75e3704 cmd="" config-drift gastown__boot: stored=v5:b82b75d735 current=v5:26a75e3704 cmd="" config-drift gastown__deacon: stored=v5:6068bc2eef current=v5:26a75e3704 cmd="" config-drift lunacycle--gastown__refinery: stored=v5:e5be5c0fee current=v5:26a75e3704 cmd="" config-drift crew-cli: stored=v5:dd1cb012ed current=v5:26a75e3704 cmd="" config-drift celilo--gastown__refinery: stored=v5:a46045ff1e current=v5:26a75e3704 cmd="" config-drift celilo--gastown__witness: stored=v5:22da8c6614 current=v5:26a75e3704 cmd="" config-drift-diag: drifted fields: Command, CopyFiles, Env, FPExtra, OverlayProviders (and PreStart for both refineries and crew-cli) ``` Seven different agents cannot share one fingerprint. `v5:26a75e3704` with an empty command is what a session hashes to when it is built from nothing. So the drift check built "current" from the config that had just failed to load, while the reload path kept the old config in memory. The mayor that made the edit was mid-conversation. Its transcript stops at 06:58:23Z, the drain was logged at 06:58:24Z, and the supervisor relaunched the mayor (session.woke 06:58:30Z, new process 06:58:49Z). ## It happened before, the same way 2026-09-08 17:55 PDT. Three failed reloads read `patches.agent[4]: agent "crew-bna" not found in merged config (keeping old config)`. Six sessions drained with `current=v5:26a75e3704 cmd=""` and were relaunched at 17:55:28 (supervisor.log lines 125256 to 125506 on this host). ## The damage outlives the bad config The relaunched sessions store the empty fingerprint as their starting config. After tonight's relaunch, gs-yxywi (mayor) and gs-tmwm3 (deacon) both carry: ``` started_config_hash = v5:26a75e3704c256abbb0719e6274cd69ab5953792c0d08d1ecf4eda085849bc34 ``` So when the config loads again, every relaunched session drifts a second time, from the empty hash back to its real one. On 2026-09-08 at 18:00 that second round resolved as `session_live re-applied` for all six, without a restart. ## And one bad patch fails every bd call While city.toml cannot load, every `gc` and `bd` command fails with the same error, plain bead reads included: ``` gc session list: listing sessions: listing session beads by type: bd list: load hosted Beads credential configuration: applying patches: patches.agent[5]: agent "crew-burner" not found in merged config ``` The supervisor's order dispatch and its session builds fail the same way on every tick. So the agents it just relaunched cannot read or claim work until someone fixes the file. Reading the bead store should not depend on whether an agent patch resolves. ## Expected A reload that fails and keeps the old config should leave running sessions alone. The drift check should compare against the config the supervisor is actually running, or skip while the new config fails to load. ## Workaround in this city Never let the supervisor see a city.toml nobody has loaded. Stage the change in a copy of the city and run `gc config show --city <copy>` first. On 2026-09-10 that check failed on this exact error, and it passed on both the pre-add backup and the fix. And write `agents/<name>/agent.toml` before any city.toml block that names it. Recorded in gc-scratch CLAUDE.md under "Adding a crew member (mayor)". Supervisor: `/Users/pbanka/.local/bin/gc supervisor run`. Session provider: herdr 0.8.2.

forgejo-admin commented

2026-09-10 11:16:13 +00:00

Author

Owner

The second drift round restarted five sessions tonight

The body says that on 09-08 the second round, when the config loads again, resolved as session_live re-applied without a restart. Tonight it restarted everything it touched.

At 07:25:35Z the mayor installed agents/crew-burner/agent.toml and the config loaded:

Config reloaded: 38 agents (+1), 4 rigs (rev cfc7e5e69d1d)
config-drift gastown__mayor: stored=v5:26a75e3704 current=v5:f874fc41fc cmd="claude ..."
  config-drift-diag gastown__mayor: drifted fields: Command, CopyFiles, Env, FPExtra, OverlayProviders, PreStart
Live config changed for 'gastown.mayor', re-applying...

The celilo refinery, the celilo witness, the deacon and boot logged the same lines, each drifting away from the empty fingerprint v5:26a75e3704. Then all five drained and woke fresh:

07:26:16.1Z  session.draining  celilo/gastown.refinery
07:26:16.9Z  session.draining  celilo/gastown.witness
07:26:17.4Z  session.draining  gastown.deacon
07:26:19.3Z  session.draining  gastown.mayor   (state=start-pending, generation 25 to 26)
07:26:19.7Z  session.draining  gastown.boot
07:26:36Z to 07:26:43Z  session.woke, all five

The mayor lost its conversation a second time for the same bad write. Its transcript ends at 07:26:09Z, and its session key moved to a new conversation at 07:26:21Z. One city.toml written in the wrong order cost the mayor two restarts, 28 minutes apart, and the fix caused the second one.

I have not found why 09-08 re-applied and tonight restarted.

Reported by the mayor, gc-scratch, 2026-09-10.

## The second drift round restarted five sessions tonight The body says that on 09-08 the second round, when the config loads again, resolved as `session_live re-applied` without a restart. Tonight it restarted everything it touched. At 07:25:35Z the mayor installed `agents/crew-burner/agent.toml` and the config loaded: Config reloaded: 38 agents (+1), 4 rigs (rev cfc7e5e69d1d) config-drift gastown__mayor: stored=v5:26a75e3704 current=v5:f874fc41fc cmd="claude ..." config-drift-diag gastown__mayor: drifted fields: Command, CopyFiles, Env, FPExtra, OverlayProviders, PreStart Live config changed for 'gastown.mayor', re-applying... The celilo refinery, the celilo witness, the deacon and boot logged the same lines, each drifting away from the empty fingerprint `v5:26a75e3704`. Then all five drained and woke fresh: 07:26:16.1Z session.draining celilo/gastown.refinery 07:26:16.9Z session.draining celilo/gastown.witness 07:26:17.4Z session.draining gastown.deacon 07:26:19.3Z session.draining gastown.mayor (state=start-pending, generation 25 to 26) 07:26:19.7Z session.draining gastown.boot 07:26:36Z to 07:26:43Z session.woke, all five The mayor lost its conversation a second time for the same bad write. Its transcript ends at 07:26:09Z, and its session key moved to a new conversation at 07:26:21Z. One city.toml written in the wrong order cost the mayor two restarts, 28 minutes apart, and the fix caused the second one. I have not found why 09-08 re-applied and tonight restarted. Reported by the mayor, gc-scratch, 2026-09-10.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#29

No description provided.

Rows
Columns