celilo/gascity

Fork 0

The reaper reads .beads/backup/backup_state.json but nothing writes it, so bulk prune never runs and it escalates every cycle while backups are healthy #22

New issue

Open

opened 2026-09-07 22:12:19 +00:00 by forgejo-admin · 2 comments

forgejo-admin commented

2026-09-07 22:12:19 +00:00

Owner

What happens

The reaper skips its bulk prune on every cycle and raises a MEDIUM escalation each time, with this message:

gm: bulk prune skipped: backup stale or absent
(source=/Users/pbanka/gc-scratch/.beads/backup/backup_state.json age=absent threshold=86400s)

The backups are fine. The file it reads is not the file anything writes.

Measured, gc-scratch, 2026-09-07 22:11Z

The path the reaper reads does not exist:

.beads/backup/backup_state.json      ABSENT

The path that IS written, and is fresh:

.beads/dolt-backup-state.json
{
  "last_sync": "2026-09-07T16:06:27Z",
  "source": "mol-dog-backup artifact /Users/pbanka/gc-scratch/.dolt-backup/hq/manifest"
}

6h old against an 86400s threshold, so comfortably inside it.

And the backups themselves are healthy. Manifest mtimes, which CLAUDE.md already names as the instrument to trust here rather than any state file:

ce    2026-09-07T22:08:46Z     <- 3 minutes old
hq    2026-09-07T22:09:00Z
lu    2026-09-07T22:09:01Z
td    2026-09-07T22:09:02Z
tn    2026-09-07T22:09:04Z
ga    2026-08-30T22:45:25Z     <- rig removed 2026-08-30, expected
byoi  2026-08-20T05:41:51Z     <- stale, separate question

Five of seven stores backed up within the last three minutes. backup_state.json appears in the gc binary's strings alongside %s: read backup_state.json: %v, so the reader is in gc, not in a script.

Why it costs something

The prune never runs. That is the actual function being skipped, silently and indefinitely, so closed beads and wisps accumulate with nothing reclaiming them. The city gets slower and nobody is told, because the message the operator sees is about backups, which are the one thing that is working.

It floods the escalation channel. 58 identical MEDIUM escalations were sitting in the mayor's inbox when this was written, and that inbox is where real escalations arrive. A false alarm at that volume is not merely noise: it trains the reader to archive on subject line, which is exactly how a genuine escalation gets missed. This is the inverse of the stale-ticket problem — not a finding that reaches nobody, but a non-finding that reaches everybody, repeatedly.

The message actively misleads. An operator who acts on it goes and investigates a backup system that is healthy. CLAUDE.md already carries an entry about bd backup status reporting "No backup has been performed yet" for stores backed up every six hours, which is a different blind instrument in the same area. That makes two reporters that disagree with reality about backups, pointing at two different state files, neither of which is the artifact.

Suggested fix

Pick one deliberately, they are not equivalent:

Read what is written. Point the reaper's freshness check at .beads/dolt-backup-state.json, the file bd backup sync actually writes.
Measure the artifact, not a state file. Take freshness from the newest .dolt-backup/<db>/manifest mtime. This is what the operator instructions already tell humans to do, and it cannot drift out of sync with the writer because there is no separate writer.
If two state files are genuinely intended, make the absence of one a startup error naming both paths, rather than a per-cycle escalation that reads as a backup problem.

Option 2 is the one that removes the class: any scheme where a reporter reads a state file some other component is responsible for writing can drift the moment either side moves, and this is the second instance of exactly that in the backup area.

Separately, the escalation should be rate-limited or deduplicated. Whatever the underlying condition, emitting the same MEDIUM 58 times is a defect on its own.

Filed from gc-scratch. No bead: no polecat in this city can change the gc binary.

## What happens The reaper skips its bulk prune on every cycle and raises a MEDIUM escalation each time, with this message: ``` gm: bulk prune skipped: backup stale or absent (source=/Users/pbanka/gc-scratch/.beads/backup/backup_state.json age=absent threshold=86400s) ``` The backups are fine. The file it reads is not the file anything writes. ## Measured, gc-scratch, 2026-09-07 22:11Z **The path the reaper reads does not exist:** ``` .beads/backup/backup_state.json ABSENT ``` **The path that IS written, and is fresh:** ``` .beads/dolt-backup-state.json { "last_sync": "2026-09-07T16:06:27Z", "source": "mol-dog-backup artifact /Users/pbanka/gc-scratch/.dolt-backup/hq/manifest" } ``` 6h old against an 86400s threshold, so comfortably inside it. **And the backups themselves are healthy.** Manifest mtimes, which `CLAUDE.md` already names as the instrument to trust here rather than any state file: ``` ce 2026-09-07T22:08:46Z <- 3 minutes old hq 2026-09-07T22:09:00Z lu 2026-09-07T22:09:01Z td 2026-09-07T22:09:02Z tn 2026-09-07T22:09:04Z ga 2026-08-30T22:45:25Z <- rig removed 2026-08-30, expected byoi 2026-08-20T05:41:51Z <- stale, separate question ``` Five of seven stores backed up within the last three minutes. `backup_state.json` appears in the gc binary's strings alongside `%s: read backup_state.json: %v`, so the reader is in gc, not in a script. ## Why it costs something **The prune never runs.** That is the actual function being skipped, silently and indefinitely, so closed beads and wisps accumulate with nothing reclaiming them. The city gets slower and nobody is told, because the message the operator sees is about *backups*, which are the one thing that is working. **It floods the escalation channel.** 58 identical MEDIUM escalations were sitting in the mayor's inbox when this was written, and that inbox is where real escalations arrive. A false alarm at that volume is not merely noise: it trains the reader to archive on subject line, which is exactly how a genuine escalation gets missed. This is the inverse of the stale-ticket problem — not a finding that reaches nobody, but a non-finding that reaches everybody, repeatedly. **The message actively misleads.** An operator who acts on it goes and investigates a backup system that is healthy. `CLAUDE.md` already carries an entry about `bd backup status` reporting "No backup has been performed yet" for stores backed up every six hours, which is a *different* blind instrument in the same area. That makes two reporters that disagree with reality about backups, pointing at two different state files, neither of which is the artifact. ## Suggested fix Pick one deliberately, they are not equivalent: 1. **Read what is written.** Point the reaper's freshness check at `.beads/dolt-backup-state.json`, the file `bd backup sync` actually writes. 2. **Measure the artifact, not a state file.** Take freshness from the newest `.dolt-backup/<db>/manifest` mtime. This is what the operator instructions already tell humans to do, and it cannot drift out of sync with the writer because there is no separate writer. 3. If two state files are genuinely intended, make the absence of one a *startup* error naming both paths, rather than a per-cycle escalation that reads as a backup problem. Option 2 is the one that removes the class: any scheme where a reporter reads a state file some other component is responsible for writing can drift the moment either side moves, and this is the second instance of exactly that in the backup area. Separately, the escalation should be rate-limited or deduplicated. Whatever the underlying condition, emitting the same MEDIUM 58 times is a defect on its own. Filed from gc-scratch. No bead: no polecat in this city can change the gc binary.

forgejo-admin commented

2026-09-07 22:14:02 +00:00

Author

Owner

A second false alarm in the same area, found in the same sweep: "Dolt backup: 1/7 databases failed to sync — hq(backup add failed)".

Also untrue, and it strengthens the case for fix option 2 rather than option 1.

Measured 2026-09-07 22:14Z, from .beads/dolt/hq:

$ dolt backup -v
hq-backup file:///Users/pbanka/gc-scratch/.dolt-backup/hq {}
exit=0

$ dolt backup add hq-backup file:///Users/pbanka/gc-scratch/.dolt-backup/hq
Error 1105 (HY000): backup 'hq-backup' already exists

So the sync job runs dolt backup add, that call fails because the remote is already registered, and the job reports it as a database that failed to sync. The sync itself is fine: .dolt-backup/hq/manifest was 3 minutes old at the time of measurement and the directory holds 2.1 GB across 86 entries.

backup add is not idempotent and the job treats its failure as a sync failure. Those are different things. Registering a remote that already exists is the normal steady state, not an error worth escalating.

Why this belongs on this issue rather than its own. Three separate reporters now disagree with reality about backups in this city, all in the same direction — claiming failure while the artifact is fresh:

the reaper's backup_state.json path (this issue),
bd backup status, which reports "No backup has been performed yet" for stores backed up every six hours (already in gc-scratch's CLAUDE.md),
the sync job's backup add handling (this comment).

Each has its own proximate cause, but the shape is identical: a reporter infers backup health from something other than the backup. Option 2 above — take freshness from the newest .dolt-backup/<db>/manifest mtime — is the only one of the three suggested fixes that closes all of them, because the manifest is the artifact and cannot disagree with itself.

The operational cost is now measurable rather than theoretical. Between this and the reaper escalation, the mayor's inbox held 58 reaper escalations plus repeated hq sync failures, all false, sitting above three genuine JSONL spike detected [HIGH] escalations showing the hq database growing 138 → 222 → 301 → 366. The real signal was underneath the noise, and the growth it reports is plausibly a consequence of the bulk prune this issue describes never running.

**A second false alarm in the same area, found in the same sweep: "Dolt backup: 1/7 databases failed to sync — hq(backup add failed)".** Also untrue, and it strengthens the case for fix option 2 rather than option 1. Measured 2026-09-07 22:14Z, from `.beads/dolt/hq`: ``` $ dolt backup -v hq-backup file:///Users/pbanka/gc-scratch/.dolt-backup/hq {} exit=0 $ dolt backup add hq-backup file:///Users/pbanka/gc-scratch/.dolt-backup/hq Error 1105 (HY000): backup 'hq-backup' already exists ``` So the sync job runs `dolt backup add`, that call fails **because the remote is already registered**, and the job reports it as a database that failed to sync. The sync itself is fine: `.dolt-backup/hq/manifest` was 3 minutes old at the time of measurement and the directory holds 2.1 GB across 86 entries. `backup add` is not idempotent and the job treats its failure as a sync failure. Those are different things. Registering a remote that already exists is the normal steady state, not an error worth escalating. **Why this belongs on this issue rather than its own.** Three separate reporters now disagree with reality about backups in this city, all in the same direction — claiming failure while the artifact is fresh: 1. the reaper's `backup_state.json` path (this issue), 2. `bd backup status`, which reports "No backup has been performed yet" for stores backed up every six hours (already in gc-scratch's `CLAUDE.md`), 3. the sync job's `backup add` handling (this comment). Each has its own proximate cause, but the shape is identical: a reporter infers backup health from something other than the backup. Option 2 above — take freshness from the newest `.dolt-backup/<db>/manifest` mtime — is the only one of the three suggested fixes that closes all of them, because the manifest is the artifact and cannot disagree with itself. The operational cost is now measurable rather than theoretical. Between this and the reaper escalation, the mayor's inbox held **58 reaper escalations plus repeated hq sync failures**, all false, sitting above three genuine `JSONL spike detected [HIGH]` escalations showing the hq database growing 138 → 222 → 301 → 366. The real signal was underneath the noise, and the growth it reports is plausibly a *consequence* of the bulk prune this issue describes never running.

forgejo-admin commented

2026-09-07 22:16:03 +00:00

Author

Owner

Root cause found, and it is upstream of everything above: bd backup sync cannot parse dolt's version string, so it refuses to run and therefore never writes the state file the reaper reads.

A fourth escalation in the same sweep, Dolt backup: dolt-too-old for backup sync [HIGH]:

Skipping backup sync: dolt version unknown is below required 2.1.0.
Gas City requires this managed Dolt floor before backup sync.

Measured on the same host, same moment:

$ dolt version
dolt version 2.3.1
$ which dolt
/opt/homebrew/bin/dolt

2.3.1 is above the 2.1.0 floor. The version is not too old. It is being read as unknown, and the comparison then fails closed.

This makes the whole chain one defect rather than three

CLAUDE.md already records that two independent pipelines write the same artifact tree: mol-dog-backup syncs every Dolt DB via the dolt CLI and stamps no state file, while bd backup sync is the sole writer of .beads/dolt-backup-state.json. That explains every observation at once:

bd backup sync misparses dolt version 2.3.1 as unknown, decides it is below the floor, and skips.
Because it skipped, it wrote no fresh state. .beads/dolt-backup-state.json is stuck at 2026-09-07T16:06:27Z and .beads/backup/backup_state.json never appears at all.
mol-dog-backup keeps running via the dolt CLI, which is why every manifest is 3 minutes old and the backups are genuinely healthy.
The reaper reads the state file the refusing pipeline would have written, finds it absent, and skips its bulk prune every cycle, escalating each time.
Closed records accumulate with nothing reclaiming them.

The consequence, measured

The hq store now holds 12,587 closed beads against 23 open (87 with wisps). That is what the three genuine JSONL spike detected [HIGH] escalations were reporting as they climbed 138 → 222 → 301 → 366.

So the prune has a real job that is not being done, and the growth is not cosmetic.

What this changes about the fix

The path mismatch in the issue body is still worth fixing, but it is the second-order problem. Fix the version parse first: bd backup sync should read dolt version 2.3.1 as 2.3.1. Whatever the parser is doing with a 2.x string, it is returning unknown rather than failing loudly, and "unknown is below required 2.1.0" is a comparison against a value that was never obtained.

Two hardening notes while that is being touched:

A version that cannot be parsed should be a loud, named error ("could not parse dolt version 2.3.1"), not silently coerced to a sentinel that then loses a comparison. The current behaviour is indistinguishable from a genuinely old dolt, which is what made this take four escalations to unpick.
The escalations should be deduplicated. One misparse produced 58 reaper escalations plus repeated sync-failure and version escalations, which buried the three real JSONL-growth alerts underneath them.

**Root cause found, and it is upstream of everything above: `bd backup sync` cannot parse dolt's version string, so it refuses to run and therefore never writes the state file the reaper reads.** A fourth escalation in the same sweep, `Dolt backup: dolt-too-old for backup sync [HIGH]`: ``` Skipping backup sync: dolt version unknown is below required 2.1.0. Gas City requires this managed Dolt floor before backup sync. ``` Measured on the same host, same moment: ``` $ dolt version dolt version 2.3.1 $ which dolt /opt/homebrew/bin/dolt ``` 2.3.1 is above the 2.1.0 floor. The version is not too old. It is being read as **`unknown`**, and the comparison then fails closed. ## This makes the whole chain one defect rather than three `CLAUDE.md` already records that two independent pipelines write the same artifact tree: `mol-dog-backup` syncs every Dolt DB via the `dolt` CLI and stamps no state file, while `bd backup sync` is the sole writer of `.beads/dolt-backup-state.json`. That explains every observation at once: 1. `bd backup sync` misparses `dolt version 2.3.1` as `unknown`, decides it is below the floor, and **skips**. 2. Because it skipped, it wrote no fresh state. `.beads/dolt-backup-state.json` is stuck at `2026-09-07T16:06:27Z` and `.beads/backup/backup_state.json` never appears at all. 3. `mol-dog-backup` keeps running via the `dolt` CLI, which is why every manifest is 3 minutes old and the backups are genuinely healthy. 4. The reaper reads the state file the refusing pipeline would have written, finds it absent, and **skips its bulk prune every cycle**, escalating each time. 5. Closed records accumulate with nothing reclaiming them. ## The consequence, measured The hq store now holds **12,587 closed beads** against 23 open (87 with wisps). That is what the three genuine `JSONL spike detected [HIGH]` escalations were reporting as they climbed 138 → 222 → 301 → 366. So the prune has a real job that is not being done, and the growth is not cosmetic. ## What this changes about the fix The path mismatch in the issue body is still worth fixing, but it is the second-order problem. **Fix the version parse first**: `bd backup sync` should read `dolt version 2.3.1` as 2.3.1. Whatever the parser is doing with a 2.x string, it is returning `unknown` rather than failing loudly, and "unknown is below required 2.1.0" is a comparison against a value that was never obtained. Two hardening notes while that is being touched: - A version that cannot be parsed should be a **loud, named error** ("could not parse `dolt version 2.3.1`"), not silently coerced to a sentinel that then loses a comparison. The current behaviour is indistinguishable from a genuinely old dolt, which is what made this take four escalations to unpick. - The escalations should be deduplicated. One misparse produced 58 reaper escalations plus repeated sync-failure and version escalations, which buried the three real JSONL-growth alerts underneath them.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#22

No description provided.

Rows
Columns