A bead lease expires under a live agent, because nothing heartbeats the bead during a long e2e suite — so `bd reclaim` robs working agents #14

New issue

Open

opened 2026-09-05 17:25:52 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-05 17:25:52 +00:00

Owner

bd reclaim exists so an abandoned claim can be taken without stealing live work, and its guard is the lease. That guard does not hold for any agent running a long e2e suite.

Nothing heartbeats the bead while the work runs. cele2e heartbeats its own flock — cele2e status reports a holder's heartbeat age in seconds, and that is what stops two sessions colliding on the rig. The bead lease is a separate clock that nobody touches for the duration. A suite outlives it, and the bead then looks abandoned while the agent is mid-run.

Measured 2026-09-05 by gastown.mayor

bead	lease	reality
`ce-yjx`	expired 09:00:32Z, eight hours earlier	`gastown.furiosa` running nine e2e suites on it, flock heartbeat 20 seconds old
`ce-ywix`	an hour past expiry	`capable` live on it

A bd reclaim at that moment would have stripped two working agents mid-run. The mayor nearly ran one, on the strength of the lease view, while clearing a genuinely orphaned claim on a third bead.

Why the two clocks disagree, and why that is the bug

The lease is sized for "an agent claimed this and went away". A cele2e census is hours of legitimate work on one bead. Those are indistinguishable to the lease and distinguishable to anything that looks at the flock, so the reaper is reading the one clock that cannot tell them apart.

The genuinely orphaned case is real and does need clearing — ce-ashq sat in_progress for 34 hours under session gs-wisp-ygccljx after that session died, and the live refinery is gs-wisp-45y3e77. So the answer is not "never reclaim"; it is that expiry alone is not evidence of abandonment.

Scope

The bead lease and whatever consumes it for reclaim/reaping in the gc source.
packages/e2e/src/run-lock.ts in celilo/celilo — the flock heartbeat that does know the agent is alive, including its SUSPECT state for a live PID that has stopped beating. That is the signal the lease view lacks.
gc session list — session liveness is a third signal, and the one that correctly identifies ce-ashq as abandoned.

Acceptance

A reclaim refuses, or warns loudly, when the claiming agent is demonstrably alive — by session liveness, by an e2e flock heartbeat, or by anything better than lease expiry alone.
The lease view itself distinguishes "expired and the holder is gone" from "expired and the holder is beating". A single expired column that means both is what nearly caused this.
Long-running work heartbeats its bead, or the lease is not used as an abandonment signal for work that can legitimately outlive it.
Recurrence gate: with a live flock heartbeat and an expired lease, reclaim must not proceed silently. Prove it by planting both states, not by reading the code — the failure here is precisely that the obvious reading of "expired" is wrong.

The orphaned claim this was found beside: ce-ashq, dead session gs-wisp-ygccljx, cleared by setting status open then assigning, which bd allows where a straight reassign is refused.
gc bd update --force is not passed through by the gc wrapper, so the documented escape from a refused reassign is unavailable through it. Worth deciding whether that is intentional.

Measurement by gastown.mayor; filed by oscar (crew-e2e) at their note that it is worth a bead of its own.

`bd reclaim` exists so an abandoned claim can be taken without stealing live work, and its guard is the lease. That guard does not hold for any agent running a long e2e suite. **Nothing heartbeats the bead while the work runs.** `cele2e` heartbeats its own flock — `cele2e status` reports a holder's heartbeat age in seconds, and that is what stops two sessions colliding on the rig. The bead lease is a separate clock that nobody touches for the duration. A suite outlives it, and the bead then looks abandoned while the agent is mid-run. ## Measured 2026-09-05 by gastown.mayor | bead | lease | reality | |---|---|---| | `ce-yjx` | expired **09:00:32Z**, eight hours earlier | `gastown.furiosa` running **nine** e2e suites on it, flock heartbeat **20 seconds** old | | `ce-ywix` | an hour past expiry | `capable` live on it | A `bd reclaim` at that moment would have stripped two working agents mid-run. The mayor nearly ran one, on the strength of the lease view, while clearing a genuinely orphaned claim on a third bead. ## Why the two clocks disagree, and why that is the bug The lease is sized for "an agent claimed this and went away". A cele2e census is hours of legitimate work on one bead. Those are indistinguishable to the lease and distinguishable to anything that looks at the flock, so the reaper is reading the one clock that cannot tell them apart. The genuinely orphaned case is real and does need clearing — `ce-ashq` sat `in_progress` for 34 hours under session `gs-wisp-ygccljx` after that session died, and the live refinery is `gs-wisp-45y3e77`. So the answer is not "never reclaim"; it is that expiry alone is not evidence of abandonment. ## Scope - The bead lease and whatever consumes it for reclaim/reaping in the `gc` source. - `packages/e2e/src/run-lock.ts` in `celilo/celilo` — the flock heartbeat that *does* know the agent is alive, including its `SUSPECT` state for a live PID that has stopped beating. That is the signal the lease view lacks. - `gc session list` — session liveness is a third signal, and the one that correctly identifies `ce-ashq` as abandoned. ## Acceptance - [ ] A reclaim refuses, or warns loudly, when the claiming agent is demonstrably alive — by session liveness, by an e2e flock heartbeat, or by anything better than lease expiry alone. - [ ] The lease view itself distinguishes "expired and the holder is gone" from "expired and the holder is beating". A single `expired` column that means both is what nearly caused this. - [ ] Long-running work heartbeats its bead, or the lease is not used as an abandonment signal for work that can legitimately outlive it. - [ ] Recurrence gate: with a live flock heartbeat and an expired lease, reclaim must not proceed silently. Prove it by planting both states, not by reading the code — the failure here is precisely that the obvious reading of "expired" is wrong. ## Related - The orphaned claim this was found beside: `ce-ashq`, dead session `gs-wisp-ygccljx`, cleared by setting status open then assigning, which `bd` allows where a straight reassign is refused. - `gc bd update --force` is not passed through by the `gc` wrapper, so the documented escape from a refused reassign is unavailable through it. Worth deciding whether that is intentional. Measurement by gastown.mayor; filed by oscar (crew-e2e) at their note that it is worth a bead of its own.

forgejo-admin referenced this issue

2026-09-21 04:37:52 +00:00

A recurring order is disabled forever by its own prior instance: an expired lease is never reclaimed, and gc order check reads "ready" for 185 hours while the order never fires #32

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#14

No description provided.

Rows
Columns

A bead lease expires under a live agent, because nothing heartbeats the bead during a long e2e suite — so bd reclaim robs working agents #14

Measured 2026-09-05 by gastown.mayor

Why the two clocks disagree, and why that is the bug

Scope

Acceptance

Related

A bead lease expires under a live agent, because nothing heartbeats the bead during a long e2e suite — so `bd reclaim` robs working agents #14