celilo/gascity

Fork 0

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and `gc order check` reads "ready" for 185 hours while the order never fires #32

New issue

Open

opened 2026-09-21 04:37:52 +00:00 by forgejo-admin · 2 comments

forgejo-admin commented

2026-09-21 04:37:52 +00:00

Owner

What happened

digest-generate (24h interval) last executed 2026-09-13T11:15:47Z. Measured 2026-09-21T04:35Z, that is 185 hours against a 24-hour interval. Before it stopped it ran daily without a miss: 09-09, 09-10, 09-11, 09-12, 09-13.

Nothing reported this. gc order check states the order is due and ready, every cycle, forever:

digest-generate   cooldown   yes   elapsed 185h18m14s >= interval 24h0m0s

That line is the whole problem in miniature: yes means "due", and the order still does not run. A reader scanning gc order check sees a ready order, not a dead one.

Mechanism

The order's 09-13 invocation created molecule wisp gs-ovijxz (mol-digest-generate), which was claimed by session gs-2s6udi (gastown.dog-2) and never completed. It sat in_progress for 7.7 days:

◐ gs-ovijxz · mol-digest-generate   [P2 · IN_PROGRESS]
Assignee: gs-2s6udi · Type: task
Created: 2026-09-13 · Started: 2026-09-13 · Updated: 2026-09-13
Lease: expires expired (heartbeat 2026-09-13)

The lease is expired and nothing reclaimed it. The order will not create a new instance while the previous one is outstanding, so one stalled molecule disables the recurring order permanently.

Why the claiming session did not finish, and why that is a second finding

gs-2s6udi is not dead. gc session list reports it active, age 7d. Its pane is a fresh Claude Code session sitting at its prompt with [Pasted text #1 +47 lines] — the molecule prompt — unsent in the input box, under an ✔ Update installed · Restart to update notice. The prompt was delivered into the pane and never submitted.

Two attempts to submit it from outside did not take: herdr pane send-keys <pane> enter, and herdr pane run <pane> ' '. The pane accepted neither, which suggests the TTY is no longer bound to a process reading input — consistent with the update notice.

So the state is: session reads active, order reads ready, lease reads expired, and nothing moves. Three surfaces, none of them wrong on its own terms, and no combination of them raises an alarm.

Impact

digest-generate produces the windowed activity digest. This city's CLAUDE.md names that digest as the correct cross-check against raw ~/.gc/supervisor.log greps, which over-answer because the log is unwindowed and holds every supervisor the city has ever run:

A count read straight off the grep says 167 and reads as a catastrophe. Read the duration= field, and compare against the digest, which windows properly and said 3 for its day.

That cross-check has been unavailable for a week while the instructions still direct agents to use it.

Workaround used

Closed gs-ovijxz with --force and a stale reason, to release the block. Whether that is sufficient to make the order fire again is being watched; if the order stays dead with no outstanding instance, the block is elsewhere and this issue should be updated.

Acceptance

An order whose previous instance holds an expired lease either reclaims it or fires a new instance; one stalled molecule cannot disable a recurring order indefinitely.
gc order check distinguishes due and will fire from due but blocked by an outstanding instance, and names the blocking bead. The current yes conflates them.
Something reports a recurring order that has missed N intervals. A daily order silent for 185 hours should not require a human to notice it in a list.

#14 — a bead lease expires under a live agent and bd reclaim robs working agents. This is the inverse failure on the same mechanism: a lease expires under a stalled agent and nothing reclaims it at all. Any fix should address both directions.
#2 — warrants routed to gastown.dog sit unclaimed. Adjacent: the dog pool again not moving work.
#28 — a session goes deaf while every surface reads healthy. Same family as the unsent-prompt half of this.

Filed with no bead, per this city's standing rule that gc defects get a Forgejo issue and no bead, because no polecat here can change the gc binary.

## What happened `digest-generate` (24h interval) last executed **2026-09-13T11:15:47Z**. Measured 2026-09-21T04:35Z, that is **185 hours** against a 24-hour interval. Before it stopped it ran daily without a miss: 09-09, 09-10, 09-11, 09-12, 09-13. Nothing reported this. `gc order check` states the order is due and ready, every cycle, forever: ``` digest-generate cooldown yes elapsed 185h18m14s >= interval 24h0m0s ``` That line is the whole problem in miniature: `yes` means "due", and the order still does not run. A reader scanning `gc order check` sees a ready order, not a dead one. ## Mechanism The order's 09-13 invocation created molecule wisp `gs-ovijxz` (`mol-digest-generate`), which was claimed by session `gs-2s6udi` (`gastown.dog-2`) and **never completed**. It sat `in_progress` for 7.7 days: ``` ◐ gs-ovijxz · mol-digest-generate [P2 · IN_PROGRESS] Assignee: gs-2s6udi · Type: task Created: 2026-09-13 · Started: 2026-09-13 · Updated: 2026-09-13 Lease: expires expired (heartbeat 2026-09-13) ``` **The lease is expired and nothing reclaimed it.** The order will not create a new instance while the previous one is outstanding, so one stalled molecule disables the recurring order permanently. ## Why the claiming session did not finish, and why that is a second finding `gs-2s6udi` is not dead. `gc session list` reports it `active`, age 7d. Its pane is a **fresh Claude Code session sitting at its prompt** with `[Pasted text #1 +47 lines]` — the molecule prompt — **unsent in the input box**, under an `✔ Update installed · Restart to update` notice. The prompt was delivered into the pane and never submitted. Two attempts to submit it from outside did not take: `herdr pane send-keys <pane> enter`, and `herdr pane run <pane> ' '`. The pane accepted neither, which suggests the TTY is no longer bound to a process reading input — consistent with the update notice. So the state is: **session reads `active`, order reads `ready`, lease reads `expired`, and nothing moves.** Three surfaces, none of them wrong on its own terms, and no combination of them raises an alarm. ## Impact `digest-generate` produces the windowed activity digest. This city's `CLAUDE.md` names that digest as the correct cross-check against raw `~/.gc/supervisor.log` greps, which over-answer because the log is unwindowed and holds every supervisor the city has ever run: > A count read straight off the grep says 167 and reads as a catastrophe. Read the `duration=` field, and compare against the digest, which windows properly and said 3 for its day. That cross-check has been unavailable for a week while the instructions still direct agents to use it. ## Workaround used Closed `gs-ovijxz` with `--force` and a stale reason, to release the block. Whether that is sufficient to make the order fire again is being watched; if the order stays dead with no outstanding instance, the block is elsewhere and this issue should be updated. ## Acceptance - [ ] An order whose previous instance holds an **expired** lease either reclaims it or fires a new instance; one stalled molecule cannot disable a recurring order indefinitely. - [ ] `gc order check` distinguishes *due and will fire* from *due but blocked by an outstanding instance*, and names the blocking bead. The current `yes` conflates them. - [ ] Something reports a recurring order that has missed N intervals. A daily order silent for 185 hours should not require a human to notice it in a list. ## Related - #14 — a bead lease expires under a *live* agent and `bd reclaim` robs working agents. This is the inverse failure on the same mechanism: a lease expires under a *stalled* agent and nothing reclaims it at all. Any fix should address both directions. - #2 — warrants routed to `gastown.dog` sit unclaimed. Adjacent: the dog pool again not moving work. - #28 — a session goes deaf while every surface reads healthy. Same family as the unsent-prompt half of this. Filed with no bead, per this city's standing rule that gc defects get a Forgejo issue and no bead, because no polecat here can change the `gc` binary.

forgejo-admin commented

2026-09-21 04:39:21 +00:00

Author

Owner

The workaround is confirmed sufficient, which pins the mechanism.

Closing the stale gs-ovijxz at 04:36Z made the order fire at 2026-09-21T04:37:44Z — its first execution since 2026-09-13T11:15:47Z, about a minute later and with no other change. So the outstanding prior instance was the entire block, and the order machinery is otherwise healthy.

That removes the alternative explanations. It was not the schedule, not the order definition, and not the executor: one molecule stuck in_progress under an expired lease disabled a daily order for 7.7 days.

Also checked, so the scope is known: a sweep of every ephemeral bead in the HQ store found 0 others in_progress (25 ephemeral visible, verified two ways because the ephemeral=true AND status=in_progress query is a known false-empty in this city). digest-generate was the only order in this state.

A follow-on condition worth separating from this issue. The revived order created gs-4qi15c (mol-digest-generate), correctly routed to gastown.dog, open and unassigned — proper pool demand. The gastown.dog pool is:

gastown.dog-2 (gs-2s6udi) — reads active, but is the wedged session described above, sitting at an unsent prompt.
gastown.dog-1 (gs-invazm) — asleep.

If the pool counts a wedged-but-active session as an occupied-and-working slot, it will not wake dog-1, and the fresh molecule sits unclaimed — the same outage with a different bead. That is being watched; if it holds, it belongs with #18 and #24 rather than here, and this comment will be updated with the result either way.

**The workaround is confirmed sufficient, which pins the mechanism.** Closing the stale `gs-ovijxz` at 04:36Z made the order fire at **2026-09-21T04:37:44Z** — its first execution since 2026-09-13T11:15:47Z, about a minute later and with no other change. So the outstanding prior instance was the entire block, and the order machinery is otherwise healthy. That removes the alternative explanations. It was not the schedule, not the order definition, and not the executor: one molecule stuck `in_progress` under an expired lease disabled a daily order for 7.7 days. Also checked, so the scope is known: a sweep of every ephemeral bead in the HQ store found **0** others `in_progress` (25 ephemeral visible, verified two ways because the `ephemeral=true AND status=in_progress` query is a known false-empty in this city). `digest-generate` was the only order in this state. **A follow-on condition worth separating from this issue.** The revived order created `gs-4qi15c` (`mol-digest-generate`), correctly routed to `gastown.dog`, open and unassigned — proper pool demand. The `gastown.dog` pool is: - `gastown.dog-2` (`gs-2s6udi`) — reads `active`, but is the wedged session described above, sitting at an unsent prompt. - `gastown.dog-1` (`gs-invazm`) — `asleep`. If the pool counts a wedged-but-`active` session as an occupied-and-working slot, it will not wake `dog-1`, and the fresh molecule sits unclaimed — the same outage with a different bead. That is being watched; if it holds, it belongs with #18 and #24 rather than here, and this comment will be updated with the result either way.

forgejo-admin commented

2026-09-21 04:54:28 +00:00

Author

Owner

Follow-on confirmed: the wedged session occupies the pool's only slot

The previous comment flagged this as a risk to watch. It held, and the supervisor log states the mechanism directly:

poolDesired: gastown.dog = 1
scaleCheck:  gastown.dog = 1

The pool's desired size is 1, and it counts wedged dog-2 (gs-2s6udi, reads active) as that one. So scaleCheck is satisfied, dog-1 (gs-invazm, asleep) is never woken, and the revived order's molecule gs-4qi15c sat unclaimed for 15+ minutes with routing correct (gc.routed_to=gastown.dog, unassigned, open).

A clean control makes this conclusive rather than suggestive. In the same window, a bead lifted into the polecat pool was claimed by celilo/gastown.furiosa in under one minute. So spawning works city-wide; what fails is specific to a pool whose only slot is held by a session that is active and not working.

That is the same family as #18 and #24 — pool bookkeeping that counts a session as productive on the strength of its liveness rather than its progress — with a new trigger: a session wedged at an unsent prompt after an in-place update.

Net effect, and why it is worse than it looks

Fixing the lease-reclaim half of this issue is necessary but not sufficient. Reclaiming gs-ovijxz would have let the order fire — as closing it by hand did — and the new molecule would then have queued behind the same occupied slot. The order recovers and the work still does not happen. Any fix should be validated against the whole path: order fires → molecule queued → claimed → completed.

Additional acceptance

A pool does not count a session that is alive-but-not-progressing toward scaleCheck. Liveness is not progress.
With poolDesired = 1 and the single occupant wedged, queued demand routed to that pool is either served by waking another member or reported as unservable. It must not sit silently.
Validated end to end: a recurring order's molecule reaches in_progress, not merely open.

Operationally in this city the unblock is to drain or restart the wedged member so the slot frees.

## Follow-on confirmed: the wedged session occupies the pool's only slot The previous comment flagged this as a risk to watch. It held, and the supervisor log states the mechanism directly: ``` poolDesired: gastown.dog = 1 scaleCheck: gastown.dog = 1 ``` The pool's desired size is **1**, and it counts wedged `dog-2` (`gs-2s6udi`, reads `active`) as that one. So `scaleCheck` is satisfied, `dog-1` (`gs-invazm`, `asleep`) is never woken, and the revived order's molecule `gs-4qi15c` sat **unclaimed for 15+ minutes** with routing correct (`gc.routed_to=gastown.dog`, unassigned, open). **A clean control makes this conclusive rather than suggestive.** In the same window, a bead lifted into the *polecat* pool was claimed by `celilo/gastown.furiosa` in **under one minute**. So spawning works city-wide; what fails is specific to a pool whose only slot is held by a session that is `active` and not working. That is the same family as #18 and #24 — pool bookkeeping that counts a session as productive on the strength of its liveness rather than its progress — with a new trigger: a session wedged at an unsent prompt after an in-place update. ### Net effect, and why it is worse than it looks Fixing the lease-reclaim half of this issue is **necessary but not sufficient**. Reclaiming `gs-ovijxz` would have let the order fire — as closing it by hand did — and the new molecule would then have queued behind the same occupied slot. The order recovers and the work still does not happen. Any fix should be validated against the whole path: order fires → molecule queued → **claimed** → completed. ### Additional acceptance - [ ] A pool does not count a session that is alive-but-not-progressing toward `scaleCheck`. Liveness is not progress. - [ ] With `poolDesired = 1` and the single occupant wedged, queued demand routed to that pool is either served by waking another member or reported as unservable. It must not sit silently. - [ ] Validated end to end: a recurring order's molecule reaches `in_progress`, not merely `open`. Operationally in this city the unblock is to drain or restart the wedged member so the slot frees.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#32

No description provided.

Rows
Columns

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32

What happened

Mechanism

Why the claiming session did not finish, and why that is a second finding

Impact

Workaround used

Acceptance

Related

Follow-on confirmed: the wedged session occupies the pool's only slot

Net effect, and why it is worse than it looks

Additional acceptance

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and `gc order check` reads "ready" for 185 hours while the order never fires #32