Wellspring network diagnosis

Wellspring (ws) diagnoses the local macOS network as a ladder of causes. Its SwiftBar surface is an ambient view of the same report used by structured output and the interactive view. A lower failing rung explains higher failures; suppressed findings stay visible and name the suppressor.

Install the diagnostic

Wellspring is the @celilo/wellspring application; its command is ws. It requires macOS and the Bun version pinned in mise.toml. Build the standalone diagnostic and its companion helper from this repository:

bun install
bun run build
sudo install -o root -g wheel -m 0755 dist/ws /usr/local/bin/ws
sudo install -o root -g wheel -m 0755 dist/ws-helper /usr/local/bin/ws-helper

Keep both compiled binaries together: helper installation locates ws-helper beside the running ws executable. bun run reinstall rebuilds and replaces both and refreshes the SwiftBar plugin when its folder is configured. If helper code changed, also repeat the explicit helper installation below; copying the companion binary does not replace the privileged installed copy. Package scripts.

WireGuard tools are optional; brew install wireguard-tools supplies wg and wg-quick when tunnel inspection is wanted. Their absence is a reported blind spot. Full fleet diagnosis also needs an installed Celilo CLI and a read-authorized remote API connection; the local diagnostic remains usable without them. Runtime code imports no @celilo/* packages, so unavailable fleet access need not prevent local diagnosis.

Read a report and inspect evidence

ws
ws report
ws --json
ws --quick
ws --menubar --quick

A bare ws opens the interactive view when stdin is a terminal; ws watch requests the same view explicitly. Report flags and non-terminal execution select static output instead. ws --menubar alone still runs the full ladder; the plugin's --quick flag is what avoids probes.

In the interactive view, arrow keys move, space opens evidence, and enter arms a remediation; a second enter confirms it after showing the inverse. Escape cancels, r probes again and q quits. Static/JSON reports exit 1 when any finding is a fault and 0 otherwise. The interactive view exits 0 on normal quit, and menu-format output exits 0 even for a fault so SwiftBar can display the finding rather than replacing it with a plugin error. Unknown flags exit 2.

The diagnostic compares link, address, gateway, overlay, resolver, resolution, transit, captive-portal, application and fleet evidence. ICMP alone proves neither a working service nor an outage. A successful connection with a failed protocol response indicates a service or policy problem; an ICMP refusal with working transport and protocol is not an outage. Probes share a bounded run budget; no response is an observation rather than an unlimited wait.

A lower fault suppresses downstream faults, naming the lowest cause rather than the nearest symptom. A healthy finding above that cause stays visible. Local firewall posture is off the traffic ladder, so normal firewall enablement cannot suppress the whole diagnosis. The default 30-second grace window controls when a persistent fault becomes actionable; it does not hide every new fault from reports. See evaluation, ladder policy, and probe policy.

Configure probes and fleet access

Optional configuration lives in ~/.config/wellspring/config.json. Missing fields use defaults; unusable values and unknown keys are reported on stderr without contaminating JSON output. A missing/unreadable file falls back to defaults. For example:

{
  "celiloEndpoint": "celilo-api@manager.example.net",
  "probeName": "example.com",
  "transitTargets": [
    {
      "host": "1.1.1.1",
      "port": 443,
      "url": "https://1.1.1.1/",
      "label": "a public resolver"
    },
    {
      "host": "example.com",
      "port": 443,
      "url": "https://example.com/",
      "label": "by name"
    }
  ],
  "captiveUrl": "http://captive.apple.com/hotspot-detect.html",
  "captiveExpect": "Success",
  "egressUrl": "https://api.ipify.org",
  "tlsDestinations": ["https://example.com"],
  "runBudgetMs": 30000,
  "graceWindowMs": 30000,
  "staleHandshakeSec": 300
}

Use your resolvable manager name or SSH alias for celiloEndpoint. The shipped default is the bare celilo-api@celilo-mgr; that name need not resolve from your laptop. Probe destinations are configurable examples, not canonical addresses for the operator's network. Retain both an IP-addressed transit target and a named one to separate name resolution from reachability. The configuration module defines current defaults.

The fleet rung runs only alerts list --json over Celilo's remote API and caches its result. Enroll a local public key on the manager with the read grant, and select its corresponding private key through your SSH configuration:

celilo api grant wellspring --key <public-key-file> --can alerts:list

Wellspring sends no mutation to the fleet. Missing CLI, unreachable manager, timeout or unreadable response makes fleet information unknown with cache-age context, rather than green or a reason to discard local diagnosis. The quick tier does not make that fleet request. See fleet adapter and Celilo CLI access.

Install the privileged helper

The diagnostic can run without a privileged helper. To enable its restricted root checks and reversible changes, first inspect the proposed installation:

sudo ws install-helper
sudo ws install-helper --commit

The first command prints the plan and does nothing. The second explicitly applies it. The installer validates sudoers content before placing it. It installs /usr/local/libexec/ws-helper as root:wheel mode 0755 and a root-owned mode-0440 entry in /etc/sudoers.d/wellspring. The entry names the helper without argument wildcards; argument restrictions live in the helper.

The closed verbs are dns-set for a named service's resolvers, vpn-up and vpn-down for a known tunnel, and the read-only pf-show and wg-show. There is no arbitrary firewall-rule loading verb. Calls use argument arrays without a shell and validate arguments against both syntax and live system state. WireGuard executable locations are a fixed Homebrew-prefix allowlist, not an operator-controlled config path or PATH search. The installed helper must remain root-owned and non-writable by its caller; the diagnostic checks that before each privileged use.

This creates a persistent passwordless-root path whose boundary is the helper's validation and ownership. Read the printed plan before committing it. Without that helper, privileged packet-filter and handshake evidence stays unknown; missing evidence must not become a clean result. See helper installer, validation, and ownership guard.

Menu bar coverage

After installing ws, install SwiftBar and copy its plugin from apps/wellspring, setting SWIFTBAR_PLUGIN_FOLDER to the folder SwiftBar actually uses:

brew install --cask swiftbar
cp swiftbar/wellspring.1m.sh "$SWIFTBAR_PLUGIN_FOLDER/"
chmod +x "$SWIFTBAR_PLUGIN_FOLDER/wellspring.1m.sh"

The SwiftBar plugin invokes ws --menubar --quick every minute and on opening. These refreshes read local system state; they issue no off-host ICMP, transport or protocol probe. Configuration reasoning remains available without probing, including a resolver configured differently from DHCP. No finding for an unrun rung may imply it passed. Choose Run the full ladder explicitly when network probing is needed.

The shared report records evaluated and skipped rungs. Faults and warnings take precedence; a clean quick result has a partial-coverage icon distinct from a full/recent clean result. A full ladder completion remains recent for ten minutes. The menu reports its age, or says it has never completed. This is coverage, not proof that off-host services stayed healthy throughout that time.

The menu currently prints evaluated/skipped counts and last-full age, while the static report names the rungs. Naming exactly which rungs were not evaluated remains an unmet menu requirement (#1710): counts alone cannot tell an operator whether transit, captive, app, resolution or fleet was checked.

The renderer consumes findings, suppression, root cause and remediation choices from the shared Report. It must not derive a competing diagnosis. The plugin locates an installed ws rather than a checkout and checks that output contains SwiftBar's menu separator; missing or old binaries produce a legible menu error.

The icon shapes are ✗ for an unsuppressed fault, ! for a warning or unknown evidence, ● for a clean full/recent evaluation, and ◐ for clean partial coverage. Shape carries the distinction when a menu theme renders monochrome.

Reversible remediation

A menu remediation is selectable only when its inverse can be captured. Before any mutation, that inverse must be persisted. Failure to write the undo record refuses the action; unreadable prior state yields a command for the operator to run manually instead of a button. Undo last action uses the recorded previous value, including reversing tunnel up/down operations.

The shared ↯ marker means a captured inverse; ✎ means a command to run manually. ws remediate <finding-key> invokes the same action path, and ws undo reverses the most recent persisted action. Only the interactive view uses the two-enter confirmation; a selectable menu item acts on its click.

The shared action code supplies the same guard to menu and interactive surfaces. Privileged actions verify the helper's ownership and mode on every call. The helper executes a closed set of verbs without a shell, and no menu click should bypass the action/undo path.

Observations and storage

Periodic evaluation records observations without changing network settings. A fault present across the grace window can therefore become actionable even when no manual report was run. The menu must lead with the same root cause as the equivalent structured report.

The findings store, cache and undo stack use temporary-file replacement so concurrent readers see a complete old or new document. This prevents partial JSON; it does not serialize concurrent read-modify-write snapshots or provide a power-loss durability guarantee. The earlier exact claim that at most one observation is lost needs concurrency evidence or stronger storage semantics. WELLSPRING_STATE_DIR selects isolated state for tests; the ordinary state directory is ~/.local/state/wellspring/.

The files are mode 0600 and hold first/last observations, acknowledgement, silence/resolve records, cached fleet data and the undo stack. Removing them also discards that history and the stored way back for earlier actions; it is not an equivalent diagnostic cleanup to preserving the existing records.

Inspect WireGuard tunnel evidence

The WireGuard macOS app's Network Extension tunnels can appear connected in scutil --nc list while wg show sees no peers. That absence is a blind spot, not proof there is no tunnel. wg-quick configurations expose handshake and transfer evidence to the privileged helper.

If moving an existing tunnel to wg-quick, export its configuration from the app, protect the exported private key, and install one tunnel at a time:

sudo install -m 600 -o root <name>.conf /etc/wireguard/<name>.conf
sudo wg-quick up <name>
sudo wg show

Keep the interface filename within 15 characters. Verify peer, handshake and transfer evidence before removing the app's original profile. Toggling now needs privilege, and wg-quick alone does not replace the app's on-demand or reboot reconnection behavior. Overlapping AllowedIPs across simultaneous tunnels can compete for routes. A running wg-quick DNS monitor can reapply its DNS configuration over manual edits; bring the tunnel down before changing those settings. A literal endpoint can become stale when the provider's address changes. The historical WireGuard rollout record retains the original migration context; this guide does not verify your tunnel.

Inspect the underlying macOS signals

For resolver precedence, use scutil --dns; ipconfig getsummary <iface> reports DHCP-offered DNS, and networksetup -getdnsservers <service> reports explicit resolver settings. networksetup -listnetworkserviceorder maps service names to interfaces. A resolver disagreement is useful evidence even without sending a probe. scutil --nc list covers GUI-managed VPN state that wg show cannot expose.

For local path evidence, netstat -rn -f inet shows routes, route -n get default identifies the gateway, and arp -n <gateway> checks neighbor resolution. In ifconfig <iface>, inspect carrier status rather than assuming the UP flag means a working physical link. socketfilterfw --getglobalstate reads application-firewall posture; privileged pfctl -si -sr reads packet-filter state. Inaccessible privileged state is unknown.

For service evidence, dig distinguishes DNS response codes and TCP/UDP; nc -z supplies a transport observation, and curl can verify protocol content and TLS. nc -n is appropriate for literals and disables hostname lookup. Treat ping as one observation alongside transport and protocol results. Wellspring uses base macOS commands plus optional WireGuard tools; it does not require socat, nmap or mtr. Inspect the collector and probe implementation when host output differs.

Validate a Wellspring change

From the repository root, use the bounded test and check commands:

bun run check
bun run build

The compiled binary is a separate deliverable from a passing source test. When a smoke run is needed, use isolated WELLSPRING_STATE_DIR and record whether the invocation used local reads or network probes. Do not treat an unperformed live diagnostic as verified behavior.

Inspected quick-tier tests, coverage tests, observation tests, and atomic-write tests establish their assertion boundaries in source. No local network inspection or remediation was run for this explanation.