One rig with an uninitialized beads store makes the demand read PARTIAL for the whole city, and every pool stops spawning while `gc status` reads healthy #24

New issue

Open

opened 2026-09-09 01:10:44 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-09 01:10:44 +00:00

Owner

Summary

A rig registered in city.toml whose .beads/ was never created fails its scale_check. That failure is not contained to the broken rig: it sets a PARTIAL flag on the demand read, and buildDesiredState refuses fresh pool creates on a partial read. So one misconfigured rig freezes polecat spawning for every rig in the city.

The interlock itself is correct. Refusing to spawn when you cannot see demand is the safe choice. The defect is the blast radius: one rig's broken store should not blind the reader for rigs whose stores are fine.

Measured

gc-scratch, 2026-09-08 18:00:25 PDT. A fifth rig bna-yard-sale-2 was added to city.toml by hand (its config appended without gc rig add), so its beads database was never created. gc rig list states it plainly, and it is the only rig in that state:

  bna-yard-sale-2:
    Path:   /Users/pbanka/hobby/bna-yard-sale-2
    Prefix: bys2
    Beads:  not initialized

~/.gc/supervisor.log, from the config reload at 18:00:25 onward — 430 occurrences of the first line in about five minutes:

buildDesiredState: default scale_check rig:bna-yard-sale-2 templates=...: Ready(): bd ready: exit status 1: Error: no beads database found

scaleCheck: PARTIAL — scale_check failed for bna-yard-sale-2/claude,
  bna-yard-sale-2/core.control-dispatcher, bna-yard-sale-2/gastown.polecat,
  bna-yard-sale-2/gastown.refinery, bna-yard-sale-2/gastown.witness,
  bna-yard-sale-2/pi, celilo/core.control-dispatcher, core.control-dispatcher,
  lunacycle/core.control-dispatcher, tango-dj/core.control-dispatcher,
  tango-nexus/core.control-dispatcher, retaining affected sessions

buildDesiredState: pool "celilo/gastown.polecat" request: pool session create
  skipped: demand read partial (partial demand read, fresh create blocked)

Note the third and fourth entries in the PARTIAL list. celilo/core.control-dispatcher and the city-level core.control-dispatcher are named as failed, and neither has anything to do with bna-yard-sale-2.

The consequence, at the same moment:

poolDesired:  celilo/gastown.polecat = 2
scaleCheck:   celilo/gastown.polecat = 2

Two seats wanted, zero sessions alive. gc session list showed no celilo/gastown.{furiosa,nux,slit,rictus,capable} at all, while gc bd ready for gc.routed_to=celilo/gastown.polecat returned seven claimable beads, two of them P1.

Why this is hard to see

Every surface reads healthy or ambiguous:

gc status prints the pool as scaled (min=0, max=5) with each seat unknown (partial status). Nothing says "blocked".
gc bd ready returns the real queue, so routing verification passes. The mayor's standard check ("is it claimable at the receiving end?") answers yes and is still wrong about whether anyone will claim it.
The only statement of the truth is the fresh create blocked line in ~/.gc/supervisor.log, which is unwindowed and carries every supervisor the city has ever run.

This is the same shape as #7: real pool demand, no spawn, one log line as the sole witness. Different cause, same silence.

Suggested fix

Scope the failure to the rig that produced it. A scale_check that cannot read rig A's store should mark rig A's templates partial and leave rig B's demand read authoritative. As written, one unreadable store degrades the read for templates that were queried successfully.

Two smaller things that would each have caught this on their own:

gc doctor should fail when a registered, unsuspended rig has no initialized beads store. gc rig list already computes and prints Beads: not initialized, so the fact is available and nothing acts on it.
Refusing to spawn deserves a louder surface than a log line. gc status knows the pool is desired at 2 and running 0; it could say why.

Workaround

gc rig suspend bna-yard-sale-2 removes it from the reconciler's set. Applied here 2026-09-09 01:1xZ. The proper repair is to register the directory through the supported path (gc rig add --adopt, or remove and re-add) so it gets a real Dolt store rather than a hand-built one.

## Summary A rig registered in `city.toml` whose `.beads/` was never created fails its `scale_check`. That failure is not contained to the broken rig: it sets a PARTIAL flag on the demand read, and `buildDesiredState` refuses fresh pool creates on a partial read. So one misconfigured rig freezes polecat spawning for **every** rig in the city. The interlock itself is correct. Refusing to spawn when you cannot see demand is the safe choice. The defect is the blast radius: one rig's broken store should not blind the reader for rigs whose stores are fine. ## Measured `gc-scratch`, 2026-09-08 18:00:25 PDT. A fifth rig `bna-yard-sale-2` was added to `city.toml` by hand (its config appended without `gc rig add`), so its beads database was never created. `gc rig list` states it plainly, and it is the only rig in that state: ``` bna-yard-sale-2: Path: /Users/pbanka/hobby/bna-yard-sale-2 Prefix: bys2 Beads: not initialized ``` `~/.gc/supervisor.log`, from the config reload at 18:00:25 onward — 430 occurrences of the first line in about five minutes: ``` buildDesiredState: default scale_check rig:bna-yard-sale-2 templates=...: Ready(): bd ready: exit status 1: Error: no beads database found scaleCheck: PARTIAL — scale_check failed for bna-yard-sale-2/claude, bna-yard-sale-2/core.control-dispatcher, bna-yard-sale-2/gastown.polecat, bna-yard-sale-2/gastown.refinery, bna-yard-sale-2/gastown.witness, bna-yard-sale-2/pi, celilo/core.control-dispatcher, core.control-dispatcher, lunacycle/core.control-dispatcher, tango-dj/core.control-dispatcher, tango-nexus/core.control-dispatcher, retaining affected sessions buildDesiredState: pool "celilo/gastown.polecat" request: pool session create skipped: demand read partial (partial demand read, fresh create blocked) ``` Note the third and fourth entries in the PARTIAL list. `celilo/core.control-dispatcher` and the city-level `core.control-dispatcher` are named as failed, and neither has anything to do with `bna-yard-sale-2`. The consequence, at the same moment: ``` poolDesired: celilo/gastown.polecat = 2 scaleCheck: celilo/gastown.polecat = 2 ``` Two seats wanted, zero sessions alive. `gc session list` showed no `celilo/gastown.{furiosa,nux,slit,rictus,capable}` at all, while `gc bd ready` for `gc.routed_to=celilo/gastown.polecat` returned seven claimable beads, two of them P1. ## Why this is hard to see Every surface reads healthy or ambiguous: - `gc status` prints the pool as `scaled (min=0, max=5)` with each seat `unknown (partial status)`. Nothing says "blocked". - `gc bd ready` returns the real queue, so routing verification passes. The mayor's standard check ("is it claimable at the receiving end?") answers yes and is still wrong about whether anyone will claim it. - The only statement of the truth is the `fresh create blocked` line in `~/.gc/supervisor.log`, which is unwindowed and carries every supervisor the city has ever run. This is the same shape as #7: real pool demand, no spawn, one log line as the sole witness. Different cause, same silence. ## Suggested fix Scope the failure to the rig that produced it. A `scale_check` that cannot read rig A's store should mark rig A's templates partial and leave rig B's demand read authoritative. As written, one unreadable store degrades the read for templates that were queried successfully. Two smaller things that would each have caught this on their own: 1. `gc doctor` should fail when a registered, unsuspended rig has no initialized beads store. `gc rig list` already computes and prints `Beads: not initialized`, so the fact is available and nothing acts on it. 2. Refusing to spawn deserves a louder surface than a log line. `gc status` knows the pool is desired at 2 and running 0; it could say why. ## Workaround `gc rig suspend bna-yard-sale-2` removes it from the reconciler's set. Applied here 2026-09-09 01:1xZ. The proper repair is to register the directory through the supported path (`gc rig add --adopt`, or remove and re-add) so it gets a real Dolt store rather than a hand-built one.

forgejo-admin referenced this issue

2026-09-09 02:11:05 +00:00

Supervisor drains a polecat within a minute of its claim: assigned work is read from the beads cache, demand from the store, so a fresh claim is invisible to one and already gone from the other for one reconcile cadence #18

forgejo-admin referenced this issue

2026-09-21 04:39:21 +00:00

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32

forgejo-admin referenced this issue

2026-09-21 04:54:28 +00:00

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32