Session reconciler retries a failing pre_start forever: 7,100 full worktree checkouts over three days against one stale ref lock, and every surface read healthy #17

New issue

Open

opened 2026-09-06 10:01:47 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-06 10:01:47 +00:00

Owner

What happened

In gc-scratch, the celilo polecat pool had demand for one more slot from 2026-09-03 00:02 until 2026-09-06 09:58 (UTC). Every reconciler tick the supervisor tried to start celilo--gastown__nux, and every attempt failed the same way:

session reconciler: starting celilo--gastown__nux: resuming session: herdr: running pre_start: pre_start[0]: exit status 1:
Preparing worktree (checking out 'gc-gastown.nux-6563d437cce5')
Updating files: 100% (3019/3019), done.
error: update_ref failed for ref 'HEAD': cannot lock ref 'HEAD': Unable to create '<repo>/.git/refs/heads/gc-gastown.nux-6563d437cce5.lock': File exists.
worktree-setup: failed to create worktree at .gc/worktrees/celilo/polecats/gastown.nux

The lock file was zero bytes, dated 2026-09-03 00:02, with no process holding it (lsof empty, no git process alive). Git's own message says to remove it by hand. Removing it ended the loop.

Measured: 7,102 starting celilo--gastown__nux lines in ~/.gc/supervisor.log, each one a full 3,019-file checkout followed by the same refusal. The supervisor process sat at 50 to 175 percent CPU, fseventsd at 15 to 25 percent, and colima's inotify forwarding pushed the churn into the docker VM, which ran at roughly 100 percent CPU with zero containers. On a 10-core host that is already the bottleneck for every merge gate, this was a measurable share of the load for three days.

Why nobody saw it

gc status showed the pool scaled with the slot start-pending. No error.
gc doctor passed.
gc session list showed a fresh session id per attempt (gs-zj450s, then gs-p7g3pi, ...), each start-pending create,config, which reads as a spawn in progress.
The only statement of the truth was one line per tick in supervisor.log, the same shape as celilo/gascity#7 (pool trigger worktree evidence invalid).

What would have prevented it

Back off after repeated identical pre_start failures. Three consecutive outcome=provider_error with the same error text should stop retrying every tick. Exponential backoff or a park-with-reason on the pool slot.
Do the cheap check before the expensive one. worktree-setup.sh could test for a stale .git/refs/heads/<branch>.lock (zero bytes, no holder) before checking out 3,000 files. Git refuses only after the checkout.
Surface it. gc status should show a pool slot that has failed to start N times and why. gc doctor should flag a ref lock with no holder older than a few minutes.

Remedy applied in gc-scratch

Removed the stale lock by hand on 2026-09-06 09:58Z after confirming no holder. No bead, because the gascity rig is no longer beaded; this issue is the record.

## What happened In gc-scratch, the celilo polecat pool had demand for one more slot from 2026-09-03 00:02 until 2026-09-06 09:58 (UTC). Every reconciler tick the supervisor tried to start `celilo--gastown__nux`, and every attempt failed the same way: ``` session reconciler: starting celilo--gastown__nux: resuming session: herdr: running pre_start: pre_start[0]: exit status 1: Preparing worktree (checking out 'gc-gastown.nux-6563d437cce5') Updating files: 100% (3019/3019), done. error: update_ref failed for ref 'HEAD': cannot lock ref 'HEAD': Unable to create '<repo>/.git/refs/heads/gc-gastown.nux-6563d437cce5.lock': File exists. worktree-setup: failed to create worktree at .gc/worktrees/celilo/polecats/gastown.nux ``` The lock file was zero bytes, dated 2026-09-03 00:02, with no process holding it (`lsof` empty, no git process alive). Git's own message says to remove it by hand. Removing it ended the loop. Measured: 7,102 `starting celilo--gastown__nux` lines in `~/.gc/supervisor.log`, each one a full 3,019-file checkout followed by the same refusal. The supervisor process sat at 50 to 175 percent CPU, `fseventsd` at 15 to 25 percent, and colima's inotify forwarding pushed the churn into the docker VM, which ran at roughly 100 percent CPU with zero containers. On a 10-core host that is already the bottleneck for every merge gate, this was a measurable share of the load for three days. ## Why nobody saw it - `gc status` showed the pool scaled with the slot start-pending. No error. - `gc doctor` passed. - `gc session list` showed a fresh session id per attempt (`gs-zj450s`, then `gs-p7g3pi`, ...), each `start-pending create,config`, which reads as a spawn in progress. - The only statement of the truth was one line per tick in `supervisor.log`, the same shape as celilo/gascity#7 (pool trigger worktree evidence invalid). ## What would have prevented it 1. **Back off after repeated identical pre_start failures.** Three consecutive `outcome=provider_error` with the same error text should stop retrying every tick. Exponential backoff or a park-with-reason on the pool slot. 2. **Do the cheap check before the expensive one.** `worktree-setup.sh` could test for a stale `.git/refs/heads/<branch>.lock` (zero bytes, no holder) before checking out 3,000 files. Git refuses only after the checkout. 3. **Surface it.** `gc status` should show a pool slot that has failed to start N times and why. `gc doctor` should flag a ref lock with no holder older than a few minutes. ## Remedy applied in gc-scratch Removed the stale lock by hand on 2026-09-06 09:58Z after confirming no holder. No bead, because the gascity rig is no longer beaded; this issue is the record.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#17

No description provided.

Rows
Columns