fix(health): don't alert when the isitup prober itself is unreachable #18

Merged

forgejo-admin merged 1 commit from fix/isitup-prober-noise into main

2026-07-31 07:49:01 +00:00

forgejo-admin commented

2026-07-31 07:42:48 +00:00

Owner

Coordination from the celilo alerting rollout (w1A): isitup.org is returning HTTP 522 (Cloudflare origin timeout) and looks dead rather than flaky, so every module probing it now reports failures that say nothing about the site being checked. celilo-website, celilo-registry and byoi have the same change; lunacycle lives in its own repo and needs its own.

The split

Two different facts were conflated:

fact	meaning	now
reachable prober says our site is down	real signal about us	check item, warns (unchanged)
the prober itself is unreachable	no signal at all	no check item

warn is not free: a warn health item still creates a warning-severity alert, and the route severity floor bottoms out at warning (AlertSeverity is only warning|critical), so a third-party outage pages the operator.

Changes

runPublicReachabilityCheck: the non-OK-HTTP path and the timeout/DNS/connection catch path now logger.info(...) and emit nothing. The status_code != 1 path from a reachable prober is untouched.
Tests assert both halves through the capturing logger (addCheck routes every non-pass item to logger.warn, so absence of that warn proves no item was recorded):
- unreachable prober emits no external_reachable check item (503 prober)
- reachable prober reporting us down still warns (status_code: 2)

The check and the isitup e2e simulator both stay — e2e resolves isitup.org to a simulator inside the sealed network, so the real path keeps getting exercised.

Known blind spot (documented in the source)

This is the only check that traverses the public path — public DNS → firewall DNAT → back in. page_renders resolves through the internal view straight to caddy. So while the prober is unreachable, external verification silently stops happening, including in the case where the fleet itself lost egress; under the old behaviour that produced an alert, if an uninterpretable one.

Closing that properly needs a staleness signal ("no successful external verification for N hours") backed by persisted state. That is deliberately not in this PR: a staleness alert against a permanently-dead prober would fire immediately and forever — the original noise wearing a hat. The honest sequence is a live second prober first, then staleness on top.

bun test health-check.test.ts → 15 pass.

🤖 Generated with Claude Code

Coordination from the celilo alerting rollout (`w1A`): `isitup.org` is returning HTTP 522 (Cloudflare origin timeout) and looks dead rather than flaky, so every module probing it now reports failures that say nothing about the site being checked. celilo-website, celilo-registry and byoi have the same change; lunacycle lives in its own repo and needs its own. ## The split Two different facts were conflated: | fact | meaning | now | |---|---|---| | reachable prober says our site is down | real signal about us | check item, warns (unchanged) | | the prober itself is unreachable | no signal at all | **no check item** | `warn` is not free: a warn health item still creates a warning-severity alert, and the route severity floor bottoms out at `warning` (`AlertSeverity` is only `warning|critical`), so a third-party outage pages the operator. ## Changes - `runPublicReachabilityCheck`: the non-OK-HTTP path and the timeout/DNS/connection `catch` path now `logger.info(...)` and emit nothing. The `status_code != 1` path from a reachable prober is untouched. - Tests assert both halves through the capturing logger (`addCheck` routes every non-pass item to `logger.warn`, so absence of that warn proves no item was recorded): - `unreachable prober emits no external_reachable check item` (503 prober) - `reachable prober reporting us down still warns` (`status_code: 2`) The check and the isitup e2e simulator both stay — e2e resolves `isitup.org` to a simulator inside the sealed network, so the real path keeps getting exercised. ## Known blind spot (documented in the source) This is the only check that traverses the *public* path — public DNS → firewall DNAT → back in. `page_renders` resolves through the internal view straight to caddy. So while the prober is unreachable, external verification silently stops happening, including in the case where the fleet itself lost egress; under the old behaviour that produced an alert, if an uninterpretable one. Closing that properly needs a staleness signal ("no successful external verification for N hours") backed by persisted state. That is deliberately **not** in this PR: a staleness alert against a permanently-dead prober would fire immediately and forever — the original noise wearing a hat. The honest sequence is a live second prober first, then staleness on top. `bun test health-check.test.ts` → 15 pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

forgejo-admin added 1 commit

2026-07-31 07:42:48 +00:00

fix(health): don't alert when the isitup prober itself is unreachable

pr-validate / validate (pull_request) Successful in 15s

Details

release / version (pull_request) Successful in 8s

Details

release / e2e (pull_request) Successful in 2m20s

Details

release / publish (pull_request) Has been skipped

Details

14eb166b33

Two facts were conflated: "a reachable prober says our site is down" (a real
signal about us) and "the prober itself is unreachable" (no signal at all). Both
emitted a warn check item, and warn is not free — a warn item still creates a
warning-severity alert and the route severity floor bottoms out at warning, so
it pages. isitup.org is currently returning HTTP 522, so the check has been
paging with a message that says nothing about lunacycle.

The prober-side paths (non-OK HTTP, and the timeout/DNS/connection catch) now
log and emit no check item. A reachable prober reporting status_code != 1 still
warns.

Known blind spot, documented in the source: this is the only check that
traverses the public path, so while the prober is down, external verification
silently stops — including if the fleet lost egress. Closing that needs a
staleness signal backed by persisted state, which is only worth building against
a live prober; isitup.org is not one.

Tests assert both halves via the capturing logger: no external_reachable warn on
a 503 prober, and a warn preserved when a reachable prober reports us down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

forgejo-admin merged commit 4744b6a589 into main

2026-07-31 07:49:01 +00:00

forgejo-admin referenced this pull request from a commit

2026-07-31 07:49:03 +00:00

Merge pull request 'fix(health): don't alert when the isitup prober itself is unreachable' (#18) from fix/isitup-prober-noise into main

forgejo-admin referenced this pull request from celilo/celilo

2026-07-31 07:51:14 +00:00

Repo-wide capability refactors do not reach modules that live in their own repos #461