ISS-013: mol-witness-patrol liveness map never sees town session beads; BEADS_DIR pins gc bd to the rig store and the recipe classifies dead pool sessions as asleep #4

New issue

Open

opened 2026-09-02 02:23:44 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-02 02:23:44 +00:00

Owner

Migrated from gc-scratch/issues/ISS-013-witness-liveness-map-blind-to-town-session-beads.yml on 2026-09-01. That directory is a local YAML tracker in the gc-scratch city; this repo is now the home for these defects.

id: ISS-013
title: mol-witness-patrol liveness map never sees town session beads; BEADS_DIR pins gc bd to the rig store and the recipe classifies dead pool sessions as asleep
type: bug
status: open
priority: P1
component: celilo/.beads/formulas/mol-witness-patrol.formula.toml (recover-orphaned-beads step)
created: 2026-09-01
reported_by: celilo/gastown.witness
upstream: event bead "event: orphan recovery batch ce-4f8 ce-29z ce-wt4 ce-qfz" (celilo store)
description: |
  The recover-orphaned-beads step builds its assignee liveness map from two
  sources: `gc session list --state=all --json` and a session-bead query
  (`gc bd list --type=session --label=gc:session --include-infra
  --include-gates --all --json --limit=0`), with the session-bead pass reduced
  AFTER the session list so ledger state wins. The session-bead query silently
  returned zero rows every time.

  What happened: gc rig-bound agents run with BEADS_DIR (and
  GC_BEADS_SCOPE_ROOT) exported, which pins `gc bd` and bare `bd` to the rig
  store regardless of CWD. `gc bd` answers "from the rig celilo store" and the
  session-bead list comes back `[]`. An empty result from the wrong instrument
  reads as "no session beads", not as a category error, so the map is built
  from the runtime view alone and every session resolves to its runtime state.
  The session list reported the four celilo pool workers (gs-wisp-6ycyxj,
  gs-wisp-dw07g3, gs-wisp-mml9d7, gs-wisp-fe50w9) as `asleep`, so the witness
  classified all four beads as not-orphaned and moved on.

  What that ruled out and what it pointed at: the beads were last touched
  2026-08-31T22:16Z, ~25h before the patrol. Nudging the sessions to wake them
  exposed the truth — one nudge refused with "session gs-wisp-dw07g3 is
  closed". Querying the town store directly (env -i, resolving
  /Users/pbanka/gc-scratch/.beads, prefix gs) showed all four session beads
  status=closed with metadata.state=orphaned. The runtime view was the stale
  one; the ledger was right. The recipe's own precedence rule (ledger wins)
  was correct but its ledger query never executed against the ledger.

  Consequence: four in_progress beads sat assigned to dead sessions for ~25
  hours. All four workers were spawned in one wave at 22:16Z and never ran a
  turn (last_active 0001-01-01), which is itself a second anomaly worth a
  look (spawn wave died silently; no cold-start events observed).

  Recovery performed 2026-09-01 by the witness: ce-4f8 work was already
  canonical on origin/polecat/ce-4f8 (patch-ids of the two work commits match
  the local branch exactly); ce-qfz worktree clean, tip already on origin/main;
  ce-29z and ce-wt4 never created worktrees. All four beads reopened to the
  pool with recovered=true. Worktrees for ce-4f8 and ce-qfz removed and
  pruned.
evidence:
  - `gc bd list --type=session --label=gc:session --include-infra --include-gates --all --json` from a rig-bound agent prints "answering from the rig celilo store" and returns [].
  - Same command with `env -i HOME=$HOME PATH=$PATH bd ...` from /Users/pbanka/gc-scratch/.beads resolves the gs-prefix town store and returns the session beads, including gs-wisp-6ycyxj/dw07g3/mml9d7/fe50w9 all status=closed state=orphaned.
  - `env` of a rig-bound agent shows BEADS_DIR=/Users/pbanka/hobby/celilo/.beads and GC_BEADS_SCOPE_ROOT=/Users/pbanka/hobby/celilo; unsetting GC_* alone does not clear it (BEADS_DIR wins).
  - `gc session nudge celilo--gc__design-author-1-pool` refused: "session gs-wisp-dw07g3 is closed", while `gc session list --state=all` showed the same session asleep with closed=false.
  - The four beads (ce-4f8 ce-29z ce-wt4 ce-qfz) held UpdatedAt 2026-08-31T22:16:56Z..22:26:26Z against a patrol at 2026-09-01T23:13Z.
proposed_fix: |
  Two-part fix. (1) Make the recipe's session-bead query store-explicit: run it
  against the town store deliberately (e.g. `bd -C /Users/pbanka/gc-scratch/.beads`
  or an env-cleared subshell), or give gc bd a --store=town flag, and fail the
  map build loudly when the query returns 0 rows while gc session list is
  non-empty (a non-empty town store always carries session beads). The recipe
  already has a fail-safe for an empty MAP; it needs a sibling fail-safe for
  an empty session-bead result. (2) Consider having the controller reconcile
  gc session list state with the session bead ledger, so a closed/orphaned
  ledger row never reads as asleep at the runtime layer.

Migrated from `gc-scratch/issues/ISS-013-witness-liveness-map-blind-to-town-session-beads.yml` on 2026-09-01. That directory is a local YAML tracker in the gc-scratch city; this repo is now the home for these defects. ```yaml id: ISS-013 title: mol-witness-patrol liveness map never sees town session beads; BEADS_DIR pins gc bd to the rig store and the recipe classifies dead pool sessions as asleep type: bug status: open priority: P1 component: celilo/.beads/formulas/mol-witness-patrol.formula.toml (recover-orphaned-beads step) created: 2026-09-01 reported_by: celilo/gastown.witness upstream: event bead "event: orphan recovery batch ce-4f8 ce-29z ce-wt4 ce-qfz" (celilo store) description: | The recover-orphaned-beads step builds its assignee liveness map from two sources: `gc session list --state=all --json` and a session-bead query (`gc bd list --type=session --label=gc:session --include-infra --include-gates --all --json --limit=0`), with the session-bead pass reduced AFTER the session list so ledger state wins. The session-bead query silently returned zero rows every time. What happened: gc rig-bound agents run with BEADS_DIR (and GC_BEADS_SCOPE_ROOT) exported, which pins `gc bd` and bare `bd` to the rig store regardless of CWD. `gc bd` answers "from the rig celilo store" and the session-bead list comes back `[]`. An empty result from the wrong instrument reads as "no session beads", not as a category error, so the map is built from the runtime view alone and every session resolves to its runtime state. The session list reported the four celilo pool workers (gs-wisp-6ycyxj, gs-wisp-dw07g3, gs-wisp-mml9d7, gs-wisp-fe50w9) as `asleep`, so the witness classified all four beads as not-orphaned and moved on. What that ruled out and what it pointed at: the beads were last touched 2026-08-31T22:16Z, ~25h before the patrol. Nudging the sessions to wake them exposed the truth — one nudge refused with "session gs-wisp-dw07g3 is closed". Querying the town store directly (env -i, resolving /Users/pbanka/gc-scratch/.beads, prefix gs) showed all four session beads status=closed with metadata.state=orphaned. The runtime view was the stale one; the ledger was right. The recipe's own precedence rule (ledger wins) was correct but its ledger query never executed against the ledger. Consequence: four in_progress beads sat assigned to dead sessions for ~25 hours. All four workers were spawned in one wave at 22:16Z and never ran a turn (last_active 0001-01-01), which is itself a second anomaly worth a look (spawn wave died silently; no cold-start events observed). Recovery performed 2026-09-01 by the witness: ce-4f8 work was already canonical on origin/polecat/ce-4f8 (patch-ids of the two work commits match the local branch exactly); ce-qfz worktree clean, tip already on origin/main; ce-29z and ce-wt4 never created worktrees. All four beads reopened to the pool with recovered=true. Worktrees for ce-4f8 and ce-qfz removed and pruned. evidence: - `gc bd list --type=session --label=gc:session --include-infra --include-gates --all --json` from a rig-bound agent prints "answering from the rig celilo store" and returns []. - Same command with `env -i HOME=$HOME PATH=$PATH bd ...` from /Users/pbanka/gc-scratch/.beads resolves the gs-prefix town store and returns the session beads, including gs-wisp-6ycyxj/dw07g3/mml9d7/fe50w9 all status=closed state=orphaned. - `env` of a rig-bound agent shows BEADS_DIR=/Users/pbanka/hobby/celilo/.beads and GC_BEADS_SCOPE_ROOT=/Users/pbanka/hobby/celilo; unsetting GC_* alone does not clear it (BEADS_DIR wins). - `gc session nudge celilo--gc__design-author-1-pool` refused: "session gs-wisp-dw07g3 is closed", while `gc session list --state=all` showed the same session asleep with closed=false. - The four beads (ce-4f8 ce-29z ce-wt4 ce-qfz) held UpdatedAt 2026-08-31T22:16:56Z..22:26:26Z against a patrol at 2026-09-01T23:13Z. proposed_fix: | Two-part fix. (1) Make the recipe's session-bead query store-explicit: run it against the town store deliberately (e.g. `bd -C /Users/pbanka/gc-scratch/.beads` or an env-cleared subshell), or give gc bd a --store=town flag, and fail the map build loudly when the query returns 0 rows while gc session list is non-empty (a non-empty town store always carries session beads). The recipe already has a fail-safe for an empty MAP; it needs a sibling fail-safe for an empty session-bead result. (2) Consider having the controller reconcile gc session list state with the session bead ledger, so a closed/orphaned ledger row never reads as asleep at the runtime layer. ```

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity-packs#4

No description provided.

Rows
Columns