celilo/gascity

Fork 0

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and `gc order check` reads "ready" for 185 hours while the order never fires #32

New issue

Open

opened 2026-09-21 04:37:52 +00:00 by forgejo-admin · 3 comments

forgejo-admin commented

2026-09-21 04:37:52 +00:00

Owner

What happened

digest-generate (24h interval) last executed 2026-09-13T11:15:47Z. Measured 2026-09-21T04:35Z, that is 185 hours against a 24-hour interval. Before it stopped it ran daily without a miss: 09-09, 09-10, 09-11, 09-12, 09-13.

Nothing reported this. gc order check states the order is due and ready, every cycle, forever:

digest-generate   cooldown   yes   elapsed 185h18m14s >= interval 24h0m0s

That line is the whole problem in miniature: yes means "due", and the order still does not run. A reader scanning gc order check sees a ready order, not a dead one.

Mechanism

The order's 09-13 invocation created molecule wisp gs-ovijxz (mol-digest-generate), which was claimed by session gs-2s6udi (gastown.dog-2) and never completed. It sat in_progress for 7.7 days:

◐ gs-ovijxz · mol-digest-generate   [P2 · IN_PROGRESS]
Assignee: gs-2s6udi · Type: task
Created: 2026-09-13 · Started: 2026-09-13 · Updated: 2026-09-13
Lease: expires expired (heartbeat 2026-09-13)

The lease is expired and nothing reclaimed it. The order will not create a new instance while the previous one is outstanding, so one stalled molecule disables the recurring order permanently.

Why the claiming session did not finish, and why that is a second finding

gs-2s6udi is not dead. gc session list reports it active, age 7d. Its pane is a fresh Claude Code session sitting at its prompt with [Pasted text #1 +47 lines] — the molecule prompt — unsent in the input box, under an ✔ Update installed · Restart to update notice. The prompt was delivered into the pane and never submitted.

Two attempts to submit it from outside did not take: herdr pane send-keys <pane> enter, and herdr pane run <pane> ' '. The pane accepted neither, which suggests the TTY is no longer bound to a process reading input — consistent with the update notice.

So the state is: session reads active, order reads ready, lease reads expired, and nothing moves. Three surfaces, none of them wrong on its own terms, and no combination of them raises an alarm.

Impact

digest-generate produces the windowed activity digest. This city's CLAUDE.md names that digest as the correct cross-check against raw ~/.gc/supervisor.log greps, which over-answer because the log is unwindowed and holds every supervisor the city has ever run:

A count read straight off the grep says 167 and reads as a catastrophe. Read the duration= field, and compare against the digest, which windows properly and said 3 for its day.

That cross-check has been unavailable for a week while the instructions still direct agents to use it.

Workaround used

Closed gs-ovijxz with --force and a stale reason, to release the block. Whether that is sufficient to make the order fire again is being watched; if the order stays dead with no outstanding instance, the block is elsewhere and this issue should be updated.

Acceptance

An order whose previous instance holds an expired lease either reclaims it or fires a new instance; one stalled molecule cannot disable a recurring order indefinitely.
gc order check distinguishes due and will fire from due but blocked by an outstanding instance, and names the blocking bead. The current yes conflates them.
Something reports a recurring order that has missed N intervals. A daily order silent for 185 hours should not require a human to notice it in a list.

#14 — a bead lease expires under a live agent and bd reclaim robs working agents. This is the inverse failure on the same mechanism: a lease expires under a stalled agent and nothing reclaims it at all. Any fix should address both directions.
#2 — warrants routed to gastown.dog sit unclaimed. Adjacent: the dog pool again not moving work.
#28 — a session goes deaf while every surface reads healthy. Same family as the unsent-prompt half of this.

Filed with no bead, per this city's standing rule that gc defects get a Forgejo issue and no bead, because no polecat here can change the gc binary.

## What happened `digest-generate` (24h interval) last executed **2026-09-13T11:15:47Z**. Measured 2026-09-21T04:35Z, that is **185 hours** against a 24-hour interval. Before it stopped it ran daily without a miss: 09-09, 09-10, 09-11, 09-12, 09-13. Nothing reported this. `gc order check` states the order is due and ready, every cycle, forever: ``` digest-generate cooldown yes elapsed 185h18m14s >= interval 24h0m0s ``` That line is the whole problem in miniature: `yes` means "due", and the order still does not run. A reader scanning `gc order check` sees a ready order, not a dead one. ## Mechanism The order's 09-13 invocation created molecule wisp `gs-ovijxz` (`mol-digest-generate`), which was claimed by session `gs-2s6udi` (`gastown.dog-2`) and **never completed**. It sat `in_progress` for 7.7 days: ``` ◐ gs-ovijxz · mol-digest-generate [P2 · IN_PROGRESS] Assignee: gs-2s6udi · Type: task Created: 2026-09-13 · Started: 2026-09-13 · Updated: 2026-09-13 Lease: expires expired (heartbeat 2026-09-13) ``` **The lease is expired and nothing reclaimed it.** The order will not create a new instance while the previous one is outstanding, so one stalled molecule disables the recurring order permanently. ## Why the claiming session did not finish, and why that is a second finding `gs-2s6udi` is not dead. `gc session list` reports it `active`, age 7d. Its pane is a **fresh Claude Code session sitting at its prompt** with `[Pasted text #1 +47 lines]` — the molecule prompt — **unsent in the input box**, under an `✔ Update installed · Restart to update` notice. The prompt was delivered into the pane and never submitted. Two attempts to submit it from outside did not take: `herdr pane send-keys <pane> enter`, and `herdr pane run <pane> ' '`. The pane accepted neither, which suggests the TTY is no longer bound to a process reading input — consistent with the update notice. So the state is: **session reads `active`, order reads `ready`, lease reads `expired`, and nothing moves.** Three surfaces, none of them wrong on its own terms, and no combination of them raises an alarm. ## Impact `digest-generate` produces the windowed activity digest. This city's `CLAUDE.md` names that digest as the correct cross-check against raw `~/.gc/supervisor.log` greps, which over-answer because the log is unwindowed and holds every supervisor the city has ever run: > A count read straight off the grep says 167 and reads as a catastrophe. Read the `duration=` field, and compare against the digest, which windows properly and said 3 for its day. That cross-check has been unavailable for a week while the instructions still direct agents to use it. ## Workaround used Closed `gs-ovijxz` with `--force` and a stale reason, to release the block. Whether that is sufficient to make the order fire again is being watched; if the order stays dead with no outstanding instance, the block is elsewhere and this issue should be updated. ## Acceptance - [ ] An order whose previous instance holds an **expired** lease either reclaims it or fires a new instance; one stalled molecule cannot disable a recurring order indefinitely. - [ ] `gc order check` distinguishes *due and will fire* from *due but blocked by an outstanding instance*, and names the blocking bead. The current `yes` conflates them. - [ ] Something reports a recurring order that has missed N intervals. A daily order silent for 185 hours should not require a human to notice it in a list. ## Related - #14 — a bead lease expires under a *live* agent and `bd reclaim` robs working agents. This is the inverse failure on the same mechanism: a lease expires under a *stalled* agent and nothing reclaims it at all. Any fix should address both directions. - #2 — warrants routed to `gastown.dog` sit unclaimed. Adjacent: the dog pool again not moving work. - #28 — a session goes deaf while every surface reads healthy. Same family as the unsent-prompt half of this. Filed with no bead, per this city's standing rule that gc defects get a Forgejo issue and no bead, because no polecat here can change the `gc` binary.

forgejo-admin commented

2026-09-21 04:39:21 +00:00

Author

Owner

The workaround is confirmed sufficient, which pins the mechanism.

Closing the stale gs-ovijxz at 04:36Z made the order fire at 2026-09-21T04:37:44Z — its first execution since 2026-09-13T11:15:47Z, about a minute later and with no other change. So the outstanding prior instance was the entire block, and the order machinery is otherwise healthy.

That removes the alternative explanations. It was not the schedule, not the order definition, and not the executor: one molecule stuck in_progress under an expired lease disabled a daily order for 7.7 days.

Also checked, so the scope is known: a sweep of every ephemeral bead in the HQ store found 0 others in_progress (25 ephemeral visible, verified two ways because the ephemeral=true AND status=in_progress query is a known false-empty in this city). digest-generate was the only order in this state.

A follow-on condition worth separating from this issue. The revived order created gs-4qi15c (mol-digest-generate), correctly routed to gastown.dog, open and unassigned — proper pool demand. The gastown.dog pool is:

gastown.dog-2 (gs-2s6udi) — reads active, but is the wedged session described above, sitting at an unsent prompt.
gastown.dog-1 (gs-invazm) — asleep.

If the pool counts a wedged-but-active session as an occupied-and-working slot, it will not wake dog-1, and the fresh molecule sits unclaimed — the same outage with a different bead. That is being watched; if it holds, it belongs with #18 and #24 rather than here, and this comment will be updated with the result either way.

**The workaround is confirmed sufficient, which pins the mechanism.** Closing the stale `gs-ovijxz` at 04:36Z made the order fire at **2026-09-21T04:37:44Z** — its first execution since 2026-09-13T11:15:47Z, about a minute later and with no other change. So the outstanding prior instance was the entire block, and the order machinery is otherwise healthy. That removes the alternative explanations. It was not the schedule, not the order definition, and not the executor: one molecule stuck `in_progress` under an expired lease disabled a daily order for 7.7 days. Also checked, so the scope is known: a sweep of every ephemeral bead in the HQ store found **0** others `in_progress` (25 ephemeral visible, verified two ways because the `ephemeral=true AND status=in_progress` query is a known false-empty in this city). `digest-generate` was the only order in this state. **A follow-on condition worth separating from this issue.** The revived order created `gs-4qi15c` (`mol-digest-generate`), correctly routed to `gastown.dog`, open and unassigned — proper pool demand. The `gastown.dog` pool is: - `gastown.dog-2` (`gs-2s6udi`) — reads `active`, but is the wedged session described above, sitting at an unsent prompt. - `gastown.dog-1` (`gs-invazm`) — `asleep`. If the pool counts a wedged-but-`active` session as an occupied-and-working slot, it will not wake `dog-1`, and the fresh molecule sits unclaimed — the same outage with a different bead. That is being watched; if it holds, it belongs with #18 and #24 rather than here, and this comment will be updated with the result either way.

forgejo-admin commented

2026-09-21 04:54:28 +00:00

Author

Owner

Follow-on confirmed: the wedged session occupies the pool's only slot

The previous comment flagged this as a risk to watch. It held, and the supervisor log states the mechanism directly:

poolDesired: gastown.dog = 1
scaleCheck:  gastown.dog = 1

The pool's desired size is 1, and it counts wedged dog-2 (gs-2s6udi, reads active) as that one. So scaleCheck is satisfied, dog-1 (gs-invazm, asleep) is never woken, and the revived order's molecule gs-4qi15c sat unclaimed for 15+ minutes with routing correct (gc.routed_to=gastown.dog, unassigned, open).

A clean control makes this conclusive rather than suggestive. In the same window, a bead lifted into the polecat pool was claimed by celilo/gastown.furiosa in under one minute. So spawning works city-wide; what fails is specific to a pool whose only slot is held by a session that is active and not working.

That is the same family as #18 and #24 — pool bookkeeping that counts a session as productive on the strength of its liveness rather than its progress — with a new trigger: a session wedged at an unsent prompt after an in-place update.

Net effect, and why it is worse than it looks

Fixing the lease-reclaim half of this issue is necessary but not sufficient. Reclaiming gs-ovijxz would have let the order fire — as closing it by hand did — and the new molecule would then have queued behind the same occupied slot. The order recovers and the work still does not happen. Any fix should be validated against the whole path: order fires → molecule queued → claimed → completed.

Additional acceptance

A pool does not count a session that is alive-but-not-progressing toward scaleCheck. Liveness is not progress.
With poolDesired = 1 and the single occupant wedged, queued demand routed to that pool is either served by waking another member or reported as unservable. It must not sit silently.
Validated end to end: a recurring order's molecule reaches in_progress, not merely open.

Operationally in this city the unblock is to drain or restart the wedged member so the slot frees.

## Follow-on confirmed: the wedged session occupies the pool's only slot The previous comment flagged this as a risk to watch. It held, and the supervisor log states the mechanism directly: ``` poolDesired: gastown.dog = 1 scaleCheck: gastown.dog = 1 ``` The pool's desired size is **1**, and it counts wedged `dog-2` (`gs-2s6udi`, reads `active`) as that one. So `scaleCheck` is satisfied, `dog-1` (`gs-invazm`, `asleep`) is never woken, and the revived order's molecule `gs-4qi15c` sat **unclaimed for 15+ minutes** with routing correct (`gc.routed_to=gastown.dog`, unassigned, open). **A clean control makes this conclusive rather than suggestive.** In the same window, a bead lifted into the *polecat* pool was claimed by `celilo/gastown.furiosa` in **under one minute**. So spawning works city-wide; what fails is specific to a pool whose only slot is held by a session that is `active` and not working. That is the same family as #18 and #24 — pool bookkeeping that counts a session as productive on the strength of its liveness rather than its progress — with a new trigger: a session wedged at an unsent prompt after an in-place update. ### Net effect, and why it is worse than it looks Fixing the lease-reclaim half of this issue is **necessary but not sufficient**. Reclaiming `gs-ovijxz` would have let the order fire — as closing it by hand did — and the new molecule would then have queued behind the same occupied slot. The order recovers and the work still does not happen. Any fix should be validated against the whole path: order fires → molecule queued → **claimed** → completed. ### Additional acceptance - [ ] A pool does not count a session that is alive-but-not-progressing toward `scaleCheck`. Liveness is not progress. - [ ] With `poolDesired = 1` and the single occupant wedged, queued demand routed to that pool is either served by waking another member or reported as unservable. It must not sit silently. - [ ] Validated end to end: a recurring order's molecule reaches `in_progress`, not merely `open`. Operationally in this city the unblock is to drain or restart the wedged member so the slot frees.

forgejo-admin commented

2026-09-26 16:12:51 +00:00

Author

Owner

Second instance, and the open question answered

Deacon patrol, 2026-09-26. Three data points, all measured.

1. The workaround held. digest-generate recovered and has fired daily since.

This issue left it open: "Whether that is sufficient to make the order fire again is being watched." It was sufficient.

gc order history digest-generate
digest-generate  gs-wisp-i0se7  2026-09-26T05:21:47Z
digest-generate  gs-wisp-j8ot   2026-09-25T05:21:37Z
gc order check -> cooldown: 13h12m43s remaining

2. mol-dog-stale-db was a second instance of the identical defect, dead 12.75 days, unreported.

last executed   2026-09-13T21:53:37Z   (cron 0 */4 * * *)
measured        2026-09-26T15:53Z      = 18368 minutes, ~76 missed occurrences
gc order check  mol-dog-stale-db  cron  yes  "cron: caught up missed occurrence"

The blocking molecule was gs-u5texh, matching gs-ovijxz in every particular:

◐ gs-u5texh · mol-dog-stale-db   [P2 · IN_PROGRESS]
Assignee: gs-9gybwa · bd.dog-1
Created: 2026-09-13 · Started: 2026-09-13 · Updated: 2026-09-13
Lease: expires expired (heartbeat 2026-09-13)

bd.dog-1 is stopped and no dog process was alive, so this claim was orphaned under a dead session rather than a stalled-but-live one. Against #14 that is a third variant: nothing had to rob a working agent here, the lease simply expired under a session that no longer exists, and still nothing reclaimed it.

3. Closing the instance released the order immediately, which confirms the mechanism a second time.

gc bd close gs-u5texh --force   (16:0xZ)
mol-dog-stale-db  gs-wisp-xjg8  2026-09-26T16:10:35Z   <- fired
gc order check -> cron: schedule not matched            <- healthy, waiting for its slot

So one stalled molecule disabled a 4-hourly order for 12.75 days, and one force-close restored it within minutes. The acceptance criteria stand exactly as written.

Two things worth adding to the issue

The cron wording is worse than the cooldown wording. digest-generate read cooldown yes elapsed 185h >= interval 24h, which at least exposes the gap. This one read cron yes cron: caught up missed occurrence. "Caught up" asserts the scheduler handled the miss, which is the opposite of the truth. mol-dog-stale-db is the only cron-triggered order in this city, so that phrasing is load-bearing for exactly one order and nobody had reason to distrust it.

The wisp-shaped query does not find the blocker. gc bd query 'ephemeral=true' returns nothing for gs-u5texh — it is not ephemeral. I ran that query first, read empty, and nearly reported "no outstanding instance, so the block is elsewhere," which would have sent this issue down a wrong path. What finds it:

gc bd list --include-infra --json --limit=0 \
  | jq -r '[.[]|select(.status=="in_progress")]'

One outage on 2026-09-13 left at least four claims outstanding. gs-ovijxz, gs-u5texh, and the deacon and celilo-witness patrol wisps all stopped at 2026-09-13T21:5xZ. Every recurring order sitting behind one of them has been dead since. Anyone auditing this should sweep for in_progress infra beads with expired leases rather than checking orders one at a time.

Filed as a comment rather than a new issue, per this city's rule about checking Forgejo first.

## Second instance, and the open question answered Deacon patrol, 2026-09-26. Three data points, all measured. **1. The workaround held. `digest-generate` recovered and has fired daily since.** This issue left it open: *"Whether that is sufficient to make the order fire again is being watched."* It was sufficient. ``` gc order history digest-generate digest-generate gs-wisp-i0se7 2026-09-26T05:21:47Z digest-generate gs-wisp-j8ot 2026-09-25T05:21:37Z gc order check -> cooldown: 13h12m43s remaining ``` **2. `mol-dog-stale-db` was a second instance of the identical defect, dead 12.75 days, unreported.** ``` last executed 2026-09-13T21:53:37Z (cron 0 */4 * * *) measured 2026-09-26T15:53Z = 18368 minutes, ~76 missed occurrences gc order check mol-dog-stale-db cron yes "cron: caught up missed occurrence" ``` The blocking molecule was `gs-u5texh`, matching `gs-ovijxz` in every particular: ``` ◐ gs-u5texh · mol-dog-stale-db [P2 · IN_PROGRESS] Assignee: gs-9gybwa · bd.dog-1 Created: 2026-09-13 · Started: 2026-09-13 · Updated: 2026-09-13 Lease: expires expired (heartbeat 2026-09-13) ``` `bd.dog-1` is stopped and no dog process was alive, so this claim was orphaned under a **dead** session rather than a stalled-but-live one. Against #14 that is a third variant: nothing had to rob a working agent here, the lease simply expired under a session that no longer exists, and still nothing reclaimed it. **3. Closing the instance released the order immediately, which confirms the mechanism a second time.** ``` gc bd close gs-u5texh --force (16:0xZ) mol-dog-stale-db gs-wisp-xjg8 2026-09-26T16:10:35Z <- fired gc order check -> cron: schedule not matched <- healthy, waiting for its slot ``` So one stalled molecule disabled a 4-hourly order for 12.75 days, and one force-close restored it within minutes. The acceptance criteria stand exactly as written. ### Two things worth adding to the issue **The cron wording is worse than the cooldown wording.** `digest-generate` read `cooldown yes elapsed 185h >= interval 24h`, which at least exposes the gap. This one read `cron yes cron: caught up missed occurrence`. "Caught up" asserts the scheduler *handled* the miss, which is the opposite of the truth. `mol-dog-stale-db` is the only cron-triggered order in this city, so that phrasing is load-bearing for exactly one order and nobody had reason to distrust it. **The wisp-shaped query does not find the blocker.** `gc bd query 'ephemeral=true'` returns nothing for `gs-u5texh` — it is not ephemeral. I ran that query first, read empty, and nearly reported "no outstanding instance, so the block is elsewhere," which would have sent this issue down a wrong path. What finds it: ``` gc bd list --include-infra --json --limit=0 \ | jq -r '[.[]|select(.status=="in_progress")]' ``` **One outage on 2026-09-13 left at least four claims outstanding.** `gs-ovijxz`, `gs-u5texh`, and the deacon and celilo-witness patrol wisps all stopped at 2026-09-13T21:5xZ. Every recurring order sitting behind one of them has been dead since. Anyone auditing this should sweep for in_progress infra beads with expired leases rather than checking orders one at a time. Filed as a comment rather than a new issue, per this city's rule about checking Forgejo first.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#32

No description provided.

Rows
Columns

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32

What happened

Mechanism

Why the claiming session did not finish, and why that is a second finding

Impact

Workaround used

Acceptance

Related

Follow-on confirmed: the wedged session occupies the pool's only slot

Net effect, and why it is worse than it looks

Additional acceptance

Second instance, and the open question answered

Two things worth adding to the issue

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and `gc order check` reads "ready" for 185 hours while the order never fires #32