ISS-010: Manifest-mtime backup freshness measure false-alarms on quiescent stores #2

New issue

Open

opened 2026-09-02 02:23:43 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-02 02:23:43 +00:00

Owner

Migrated from gc-scratch/issues/ISS-010-manifest-mtime-freshness-false-alarms-on-quiet-stores.yml on 2026-09-01. That directory is a local YAML tracker in the gc-scratch city; this repo is now the home for these defects.

id: ISS-010
title: Manifest-mtime backup freshness measure false-alarms on quiescent stores
type: bug
status: open
priority: P2
component: deacon patrol backup freshness check / .dolt-backup manifest mtime as freshness signal
created: 2026-08-31
reported_by: gastown.dog-1
upstream: gs-ueep
description: |
  Deacon patrol filed gs-ueep on 2026-08-31 claiming the ga store backup was
  21h stale and had "missed the 06:00Z and 15:08Z batches", because its
  .dolt-backup/ga/manifest mtime read 2026-08-30T22:45Z while ce/lu/hq
  refreshed at 15:08Z. Expected: a stale manifest means a missed backup.
  Happened: I ran the backup order scoped to ga
  (gc order run mol-dog-backup --var GC_BACKUP_DATABASES=ga). It exited 0,
  synced 1/1, and the manifest did not move. That ruled out a missed
  schedule (gc order history shows mol-dog-backup fired at 05:59:59Z and
  15:08:50Z with ga in its auto-discovered set) and ruled out a failed sync
  (manual run from the ga repo dir: exit 0). It pointed at the measure
  itself: dolt backup sync touches the manifest only when it pushes new
  data. ga's latest dolt_log commit is 2026-08-30T17:28Z, before its last
  manifest at 22:45Z, so there was nothing to push. td and tn show the same
  shape (latest commits 17:16Z on 08-30, manifests 22:45Z 08-30). ce, the
  actively written store, advances every batch. The stores the patrol
  flagged as stale are exactly the quiescent ones.

  So the manifest mtime answers "when did this store last have new data",
  not "when was this store last backed up". For a store with no writes the
  measure reads stale forever, and every patrol cycle past 24h of quiet
  produces another false escalation. gs-ueep is one instance; the same
  reading likely produced the stale td/tn/ byoi observations in ISS-006's
  evidence list. This is the false-positive twin of ISS-006's finding: that
  issue is a real skip being missed by health tooling; this issue is a
  quiet store being reported as a failure when its backup is current.
evidence:
  - gs-ueep claimed ga manifest 21h stale at 2026-08-31T19:35Z (manifest 2026-08-30T22:45:25Z)
  - gc dolt sql on ga: max(dolt_log.date) = 2026-08-30 17:28:19Z (595 commits), older than the manifest
  - gc order run mol-dog-backup --var GC_BACKUP_DATABASES=ga reported "synced: 1/1"; manifest mtime unchanged after the run
  - manual `dolt backup sync ga-backup` from the ga repo dir: exit 0, manifest mtime unchanged
  - td and tn latest commits 2026-08-30 17:16Z, both manifests 2026-08-30 22:45Z, both unflagged only because patrol scoped to ga
  - ce latest commit 2026-08-31 20:03Z, manifest advancing each batch with writes
proposed_fix: |
  Change the freshness question from "how old is the manifest" to "is there
  committed data newer than the manifest". Concretely: per store, compare
  max(dolt_log.date) against the manifest mtime (or the manifest's recorded
  commit). Stale means max(date) > manifest time, i.e. committed data the
  backup does not cover. A manifest older than the cadence with no newer
  commits is healthy quiescence and should not alarm. The deacon patrol's
  backup check, the reaper's prune gate, and stamp-backup-state.sh (which
  republishes the same mtime into dolt-backup-state.json) all read the old
  measure and need the same correction. Cross-reference ISS-006: its
  proposed per-store health check should implement this comparison, not
  bare manifest age.

Migrated from `gc-scratch/issues/ISS-010-manifest-mtime-freshness-false-alarms-on-quiet-stores.yml` on 2026-09-01. That directory is a local YAML tracker in the gc-scratch city; this repo is now the home for these defects. ```yaml id: ISS-010 title: Manifest-mtime backup freshness measure false-alarms on quiescent stores type: bug status: open priority: P2 component: deacon patrol backup freshness check / .dolt-backup manifest mtime as freshness signal created: 2026-08-31 reported_by: gastown.dog-1 upstream: gs-ueep description: | Deacon patrol filed gs-ueep on 2026-08-31 claiming the ga store backup was 21h stale and had "missed the 06:00Z and 15:08Z batches", because its .dolt-backup/ga/manifest mtime read 2026-08-30T22:45Z while ce/lu/hq refreshed at 15:08Z. Expected: a stale manifest means a missed backup. Happened: I ran the backup order scoped to ga (gc order run mol-dog-backup --var GC_BACKUP_DATABASES=ga). It exited 0, synced 1/1, and the manifest did not move. That ruled out a missed schedule (gc order history shows mol-dog-backup fired at 05:59:59Z and 15:08:50Z with ga in its auto-discovered set) and ruled out a failed sync (manual run from the ga repo dir: exit 0). It pointed at the measure itself: dolt backup sync touches the manifest only when it pushes new data. ga's latest dolt_log commit is 2026-08-30T17:28Z, before its last manifest at 22:45Z, so there was nothing to push. td and tn show the same shape (latest commits 17:16Z on 08-30, manifests 22:45Z 08-30). ce, the actively written store, advances every batch. The stores the patrol flagged as stale are exactly the quiescent ones. So the manifest mtime answers "when did this store last have new data", not "when was this store last backed up". For a store with no writes the measure reads stale forever, and every patrol cycle past 24h of quiet produces another false escalation. gs-ueep is one instance; the same reading likely produced the stale td/tn/ byoi observations in ISS-006's evidence list. This is the false-positive twin of ISS-006's finding: that issue is a real skip being missed by health tooling; this issue is a quiet store being reported as a failure when its backup is current. evidence: - gs-ueep claimed ga manifest 21h stale at 2026-08-31T19:35Z (manifest 2026-08-30T22:45:25Z) - gc dolt sql on ga: max(dolt_log.date) = 2026-08-30 17:28:19Z (595 commits), older than the manifest - gc order run mol-dog-backup --var GC_BACKUP_DATABASES=ga reported "synced: 1/1"; manifest mtime unchanged after the run - manual `dolt backup sync ga-backup` from the ga repo dir: exit 0, manifest mtime unchanged - td and tn latest commits 2026-08-30 17:16Z, both manifests 2026-08-30 22:45Z, both unflagged only because patrol scoped to ga - ce latest commit 2026-08-31 20:03Z, manifest advancing each batch with writes proposed_fix: | Change the freshness question from "how old is the manifest" to "is there committed data newer than the manifest". Concretely: per store, compare max(dolt_log.date) against the manifest mtime (or the manifest's recorded commit). Stale means max(date) > manifest time, i.e. committed data the backup does not cover. A manifest older than the cadence with no newer commits is healthy quiescence and should not alarm. The deacon patrol's backup check, the reaper's prune gate, and stamp-backup-state.sh (which republishes the same mtime into dolt-backup-state.json) all read the old measure and need the same correction. Cross-reference ISS-006: its proposed per-store health check should implement this comparison, not bare manifest age. ```

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity-packs#2

No description provided.

Rows
Columns