hold-label-routed-to check and cache-reconcile fight over gc.routed_to on hold:<role> beads: --fix writes the bare value, the cache re-stamps the bound identity within seconds, warning never clears #23

New issue

Open

opened 2026-09-07 23:20:02 +00:00 by forgejo-admin · 0 comments

forgejo-admin commented

2026-09-07 23:20:02 +00:00

Owner

Filed by gastown.deacon during a patrol on 2026-09-07.

The defect

gc doctor --fix and the cache-reconcile write path disagree on the value of gc.routed_to for beads carrying hold:<role> labels, so the hold-label-routed-to warning can never stay clear and the two planes flap against each other.

The check (cmd/gc/doctor_hold_label_routed_to.go) wants the BARE hold value: hold:mayor requires gc.routed_to == "mayor". Its Fix writes that bare value through the raw store (store.SetMetadata).

But a write of the bare value through gc bd update --set-metadata gc.routed_to=mayor is normalized on the way in: reads through the CachingStore show gastown.mayor (the bound agent identity). And the cache-reconcile actor then re-stamps the bound identity back onto the raw store.

Measured evidence (2026-09-07 22:55Z-23:15Z)

57 open celilo beads carried hold:mayor with gc.routed_to="gastown.mayor" and no bare-value form anywhere. gc doctor --fix (second run, to completion) reported 1 fixed and a green hold-label-routed-to (fixed) in-process.
gc bd update ce-0fmi --set-metadata gc.routed_to=mayor returns success. A read 8s later shows mayor. A read 24s later shows gastown.mayor again.
events.jsonl pins the actor:
- 16:13:14 actor cache-reconcile emits ce-0fmi with gc.routed_to:"mayor" (my write)
- 16:13:18 actor cache-reconcile emits ce-0fmi with gc.routed_to:"gastown.mayor" (4 seconds later)
After the fix run reported green, a fresh gc doctor warned again, count 55 (was 58). The population shifts between runs.

Why it matters

The doctor warning is a standing false alarm AND a standing real one, depending on which plane you believe. The raw store holds the check's desired value for seconds at a time. The read plane agents actually consume holds the bound identity. Every patrol that sees the warning will re-run the fix, which re-flaps the write, which re-emits events for 55+ beads. That is a permanent treadmill under a check that looks like it converged.

The route-recovery lane already has vocabulary for this exact shape (route_recovery_lane.go: "a sibling lane is clearing gc.routed_to behind us... a faster treadmill is not a fix") but that lane only restores EMPTY routes, so it is not the reverter here; the cache-reconcile re-stamp is.

What the deacon did

Ran gc doctor --fix twice (second run to completion, reported 1 fixed).
Set gc.routed_to=mayor on the affected beads once via gc bd update (normalized away).
Did NOT attempt further writes once the flap was measured. Left state as the cache plane left it (gastown.mayor).
Withheld further automated remediation; the warning will keep showing in patrols until this lands.

Suggested direction (for the owner to decide)

The two components need one contract for what a hold:<value> label implies for gc.routed_to:

If the bound identity is the contract, the check (and its fix) should resolve <value> through the same binding the CLI write path uses, and compare against the resolved form.
If the bare value is the contract, the cache write/absorb path must not normalize gc.routed_to for hold-labeled beads, and cache-reconcile must not re-stamp a value it did not observe a change for.

Option 1 looks smaller: the check already has the city config in scope, and the CLI normalization is established behavior that other consumers may depend on.

Also worth a look in the same area: cache-reconcile re-emitting a full bead snapshot 4s after an unrelated write makes every metadata diff on the city unreadable from events.jsonl, which is how this defect hid for a while (the event stream shows both values as writes from the same actor).

Filed by gastown.deacon during a patrol on 2026-09-07. # The defect `gc doctor --fix` and the cache-reconcile write path disagree on the value of `gc.routed_to` for beads carrying `hold:<role>` labels, so the `hold-label-routed-to` warning can never stay clear and the two planes flap against each other. The check (cmd/gc/doctor_hold_label_routed_to.go) wants the BARE hold value: `hold:mayor` requires `gc.routed_to == "mayor"`. Its Fix writes that bare value through the raw store (`store.SetMetadata`). But a write of the bare value through `gc bd update --set-metadata gc.routed_to=mayor` is normalized on the way in: reads through the CachingStore show `gastown.mayor` (the bound agent identity). And the cache-reconcile actor then re-stamps the bound identity back onto the raw store. # Measured evidence (2026-09-07 22:55Z-23:15Z) 1. 57 open celilo beads carried `hold:mayor` with `gc.routed_to="gastown.mayor"` and no bare-value form anywhere. `gc doctor --fix` (second run, to completion) reported `1 fixed` and a green `hold-label-routed-to (fixed)` in-process. 2. `gc bd update ce-0fmi --set-metadata gc.routed_to=mayor` returns success. A read 8s later shows `mayor`. A read 24s later shows `gastown.mayor` again. 3. events.jsonl pins the actor: - 16:13:14 actor `cache-reconcile` emits ce-0fmi with `gc.routed_to:"mayor"` (my write) - 16:13:18 actor `cache-reconcile` emits ce-0fmi with `gc.routed_to:"gastown.mayor"` (4 seconds later) 4. After the fix run reported green, a fresh `gc doctor` warned again, count 55 (was 58). The population shifts between runs. # Why it matters The doctor warning is a standing false alarm AND a standing real one, depending on which plane you believe. The raw store holds the check's desired value for seconds at a time. The read plane agents actually consume holds the bound identity. Every patrol that sees the warning will re-run the fix, which re-flaps the write, which re-emits events for 55+ beads. That is a permanent treadmill under a check that looks like it converged. The route-recovery lane already has vocabulary for this exact shape (route_recovery_lane.go: "a sibling lane is clearing gc.routed_to behind us... a faster treadmill is not a fix") but that lane only restores EMPTY routes, so it is not the reverter here; the cache-reconcile re-stamp is. # What the deacon did - Ran `gc doctor --fix` twice (second run to completion, reported `1 fixed`). - Set `gc.routed_to=mayor` on the affected beads once via `gc bd update` (normalized away). - Did NOT attempt further writes once the flap was measured. Left state as the cache plane left it (`gastown.mayor`). - Withheld further automated remediation; the warning will keep showing in patrols until this lands. # Suggested direction (for the owner to decide) The two components need one contract for what a `hold:<value>` label implies for `gc.routed_to`: 1. If the bound identity is the contract, the check (and its fix) should resolve `<value>` through the same binding the CLI write path uses, and compare against the resolved form. 2. If the bare value is the contract, the cache write/absorb path must not normalize `gc.routed_to` for hold-labeled beads, and cache-reconcile must not re-stamp a value it did not observe a change for. Option 1 looks smaller: the check already has the city config in scope, and the CLI normalization is established behavior that other consumers may depend on. Also worth a look in the same area: `cache-reconcile` re-emitting a full bead snapshot 4s after an unrelated write makes every metadata diff on the city unreadable from events.jsonl, which is how this defect hid for a while (the event stream shows both values as writes from the same actor).

forgejo-admin referenced this issue

2026-09-09 19:36:15 +00:00

Pool demand queries exclude hold:* labels only: a bead held by metadata.hold_reason reads as claimable work to every pool — measured 6 held, 2 served up by gc hook #25

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

celilo/gascity#23

No description provided.

Rows
Columns