ISS-006: Dolt backup order ran but did not back up the hq store #1

New issue

Open

opened 2026-09-02 02:23:42 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-02 02:23:42 +00:00

Owner

Migrated from gc-scratch/issues/ISS-006-backup-run-skipped-hq-store.yml on 2026-09-01. That directory is a local YAML tracker in the gc-scratch city; this repo is now the home for these defects.

id: ISS-006
title: Dolt backup order ran but did not back up the hq store
type: bug
status: open
priority: P2
component: mol-dog-backup order / .dolt-backup pipeline
created: 2026-08-30
reported_by: gastown.deacon
description: |
  The hq store is the most actively written store in the city (the HQ beads
  store holds every gs- wisp; 937 commits, 12 open beads as of this filing),
  so it should never be the stalest backup. Expected: every mol-dog-backup
  run refreshes all seven stores, or at least every store with writes since
  its last backup. Happened: the 16:45Z run on 2026-08-30 refreshed ce and
  ga (both manifests 17:02Z) but left hq at its 06:43Z manifest, which is
  exactly the execution time of the previous backup run.

  Ruled out: the backup machinery being down (ce and ga refreshed in the
  same run), and a missed schedule (mol-dog-backup fired on time at 16:45Z,
  per gc order history). That points at store selection inside the backup
  run: it picked some stores and silently skipped hq. The skip produced no
  error I could find. `gc dolt health` reported dolt_stale=false at the
  same time, so the health reporter's policy also does not see the 14h-old
  hq backup as a problem, which means nothing in the current tooling would
  ever catch a store being dropped from the rotation.

  Found by deacon patrol 2026-08-30 ~20:45Z, measuring backup age from
  .dolt-backup/<store>/manifest mtimes per the city instructions trap list.

  SECOND INSTANCE, 2026-09-01, and it inverts which store was skipped. This
  is what raises the issue from "hq was dropped" to "store selection is
  nondeterministic". The 2026-09-01T20:47Z run refreshed hq and skipped ce
  and lu, the exact opposite of the 08-30 run. Both active rigs' bead
  ledgers sat 23h behind while the store this issue was originally filed
  about was the one that got backed up.

  How that was established, since the obvious instrument is untrustworthy
  here. Manifest mtime alone cannot distinguish a skipped store from a quiet
  one (ISS-010). Applying ISS-010's own test instead: ce's latest dolt_log
  commit was 2026-09-02T01:04:40Z against a manifest of 2026-09-01T02:00:37Z,
  and lu's was 01:04:18Z against 02:00:54Z. Both had real committed data
  newer than their backup, so both were genuine skips and not quiet stores.

  What proves the ORDER RAN rather than the schedule failing: the 3h
  stamp-backup-state order wrote .beads/dolt-backup-state.json with
  last_sync 2026-09-01T20:47:23Z, sourced from the hq manifest, and hq's
  manifest carries exactly that timestamp. So mol-dog-backup executed at
  20:47Z and synced hq in that run. The schedule is healthy. An earlier
  filing of mine (ISS-015) claimed the schedule had stopped firing; that was
  wrong, it was inferred from a `gc order history` window only 4.2 hours
  deep, which cannot show a 6h order at all. ISS-015 is withdrawn and folded
  into this issue.

  Remediated by hand 2026-09-02T01:05Z: `gc order run mol-dog-backup --var
  GC_BACKUP_DATABASES=ce,lu`, exit 0, "synced: 2/2". Verified by re-reading
  both manifests (01:05:23Z and 01:05:24Z, newer than each store's latest
  commit) rather than by the receipt, because ISS-010 documents that receipt
  appearing while nothing moves.

evidence:
  - manifest mtimes at 2026-08-30T20:45Z: ce 17:02:44Z, ga 17:02:51Z, hq 06:43:26Z, lu 08-28, td/tn 08-27, byoi 08-20
  - gc order history mol-dog-backup: fired gs-wisp-bd8j6c at 16:45:03Z, previous gs-wisp-vd27bn at 06:43:24Z (matches hq manifest time exactly)
  - gc dolt health --json at 20:36Z: backups.dolt_freshness empty, dolt_age_sec 0, dolt_stale false
  - 2026-09-01 instance: ce manifest 02:00:37Z vs latest commit 2026-09-02T01:04:40Z; lu manifest 02:00:54Z vs 01:04:18Z; hq manifest 20:47:23Z (fresh)
  - .beads/dolt-backup-state.json last_sync 2026-09-01T20:47:23Z sourced from the hq manifest, proving the 20:47Z run executed and covered hq only
  - manual scoped run 2026-09-02T01:05Z exit 0 "synced: 2/2"; both manifests then advanced, so the machinery works when the stores are named explicitly
  - the two instances skip DIFFERENT stores (08-30 skipped hq, 09-01 skipped ce and lu), so the selection is not a fixed exclusion
proposed_fix: |
  UPDATED 2026-09-01. The 09-01 instance shows a scoped run works
  (GC_BACKUP_DATABASES=ce,lu synced both), so the failure is in how an
  UNSCOPED run auto-discovers and selects its store set, not in the sync. Read
  mol-dog-backup.sh's discovery step first; a partial result there would
  explain both instances and would explain why the skip is silent.

  Inspect the mol-dog-backup wisp gs-wisp-bd8j6c and the run it performed
  to see why hq was not selected. Then either fix the selection or make a
  skipped store loud (report per-store outcomes, and have gc dolt health
  flag any store whose backup age exceeds the cadence instead of reporting
  a single global dolt_stale).

Migrated from `gc-scratch/issues/ISS-006-backup-run-skipped-hq-store.yml` on 2026-09-01. That directory is a local YAML tracker in the gc-scratch city; this repo is now the home for these defects. ```yaml id: ISS-006 title: Dolt backup order ran but did not back up the hq store type: bug status: open priority: P2 component: mol-dog-backup order / .dolt-backup pipeline created: 2026-08-30 reported_by: gastown.deacon description: | The hq store is the most actively written store in the city (the HQ beads store holds every gs- wisp; 937 commits, 12 open beads as of this filing), so it should never be the stalest backup. Expected: every mol-dog-backup run refreshes all seven stores, or at least every store with writes since its last backup. Happened: the 16:45Z run on 2026-08-30 refreshed ce and ga (both manifests 17:02Z) but left hq at its 06:43Z manifest, which is exactly the execution time of the previous backup run. Ruled out: the backup machinery being down (ce and ga refreshed in the same run), and a missed schedule (mol-dog-backup fired on time at 16:45Z, per gc order history). That points at store selection inside the backup run: it picked some stores and silently skipped hq. The skip produced no error I could find. `gc dolt health` reported dolt_stale=false at the same time, so the health reporter's policy also does not see the 14h-old hq backup as a problem, which means nothing in the current tooling would ever catch a store being dropped from the rotation. Found by deacon patrol 2026-08-30 ~20:45Z, measuring backup age from .dolt-backup/<store>/manifest mtimes per the city instructions trap list. SECOND INSTANCE, 2026-09-01, and it inverts which store was skipped. This is what raises the issue from "hq was dropped" to "store selection is nondeterministic". The 2026-09-01T20:47Z run refreshed hq and skipped ce and lu, the exact opposite of the 08-30 run. Both active rigs' bead ledgers sat 23h behind while the store this issue was originally filed about was the one that got backed up. How that was established, since the obvious instrument is untrustworthy here. Manifest mtime alone cannot distinguish a skipped store from a quiet one (ISS-010). Applying ISS-010's own test instead: ce's latest dolt_log commit was 2026-09-02T01:04:40Z against a manifest of 2026-09-01T02:00:37Z, and lu's was 01:04:18Z against 02:00:54Z. Both had real committed data newer than their backup, so both were genuine skips and not quiet stores. What proves the ORDER RAN rather than the schedule failing: the 3h stamp-backup-state order wrote .beads/dolt-backup-state.json with last_sync 2026-09-01T20:47:23Z, sourced from the hq manifest, and hq's manifest carries exactly that timestamp. So mol-dog-backup executed at 20:47Z and synced hq in that run. The schedule is healthy. An earlier filing of mine (ISS-015) claimed the schedule had stopped firing; that was wrong, it was inferred from a `gc order history` window only 4.2 hours deep, which cannot show a 6h order at all. ISS-015 is withdrawn and folded into this issue. Remediated by hand 2026-09-02T01:05Z: `gc order run mol-dog-backup --var GC_BACKUP_DATABASES=ce,lu`, exit 0, "synced: 2/2". Verified by re-reading both manifests (01:05:23Z and 01:05:24Z, newer than each store's latest commit) rather than by the receipt, because ISS-010 documents that receipt appearing while nothing moves. evidence: - manifest mtimes at 2026-08-30T20:45Z: ce 17:02:44Z, ga 17:02:51Z, hq 06:43:26Z, lu 08-28, td/tn 08-27, byoi 08-20 - gc order history mol-dog-backup: fired gs-wisp-bd8j6c at 16:45:03Z, previous gs-wisp-vd27bn at 06:43:24Z (matches hq manifest time exactly) - gc dolt health --json at 20:36Z: backups.dolt_freshness empty, dolt_age_sec 0, dolt_stale false - 2026-09-01 instance: ce manifest 02:00:37Z vs latest commit 2026-09-02T01:04:40Z; lu manifest 02:00:54Z vs 01:04:18Z; hq manifest 20:47:23Z (fresh) - .beads/dolt-backup-state.json last_sync 2026-09-01T20:47:23Z sourced from the hq manifest, proving the 20:47Z run executed and covered hq only - manual scoped run 2026-09-02T01:05Z exit 0 "synced: 2/2"; both manifests then advanced, so the machinery works when the stores are named explicitly - the two instances skip DIFFERENT stores (08-30 skipped hq, 09-01 skipped ce and lu), so the selection is not a fixed exclusion proposed_fix: | UPDATED 2026-09-01. The 09-01 instance shows a scoped run works (GC_BACKUP_DATABASES=ce,lu synced both), so the failure is in how an UNSCOPED run auto-discovers and selects its store set, not in the sync. Read mol-dog-backup.sh's discovery step first; a partial result there would explain both instances and would explain why the skip is silent. Inspect the mol-dog-backup wisp gs-wisp-bd8j6c and the run it performed to see why hq was not selected. Then either fix the selection or make a skipped store loud (report per-store outcomes, and have gc dolt health flag any store whose backup age exceeds the cadence instead of reporting a single global dolt_stale). ```

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity-packs#1

No description provided.

Rows
Columns