Keep the management server's bun in lockstep with celilo: the postinst seeds the pin once, and there is no celilo route to change it #1379

New issue

Open

opened 2026-09-17 06:36:21 +00:00 by forgejo-admin · 1 comment

forgejo-admin commented

2026-09-17 06:36:21 +00:00

Owner

The celilo deb pins bun through /var/celilo/.mise.toml, but it writes that file only on first install. Every later upgrade leaves the pin alone, so the management server runs whatever bun it was first installed with, forever. There is no celilo command to change it either. The only way to move it today is to SSH in and edit the file by hand.

Measured on celilo-mgr, 2026-09-17

/var/celilo/.mise.toml          bun = "1.3.3"   written 2026-06-06 (first install)
/etc/celilo/mise.toml.template  bun = "1.4.2"   shipped by the 3.0.0 deb
mise exec -- bun --version      1.3.3

packaging/celilo/scripts/postinst:39-43:

# 3. Seed the conffile (does not clobber operator changes).
if [ ! -f "$CELILO_HOME/.mise.toml" ]; then
  install -o "$CELILO_USER" -g "$CELILO_GROUP" -m 0644 \
    /etc/celilo/mise.toml.template "$CELILO_HOME/.mise.toml"
fi

The template pinned 1.3.3 from 50fbd69c (2026-05-30) until 85e5665c moved it to 1.4.2 (2026-09-04). celilo-mgr predates the move, so upgrades through 3.0.0 all skipped the write.

Why it matters

celilo 3.0.0 was built and tested against bun 1.4.2 and was running in production on 1.3.3. These are not interchangeable: the city's own instructions record that 1.3.3's Bun.YAML.parse reads a bare on: key as true, and a merge-gate test failed on exactly that. The pin is meant to guarantee the server runs the runtime the release was tested on, and on a long-lived server it guarantees the opposite.

The guard's intent — "does not clobber operator changes" — is reasonable. The implementation cannot tell an operator's change from an untouched file an older package wrote, so it treats every existing file as operator-owned.

What was done by hand, and why that is not the fix

On 2026-09-17 around 06:21Z, crew-builder repinned the server by hand over SSH:

cp .mise.toml .mise.toml.bak-20260917
sed -i 's/^bun = "1.3.3"$/bun = "1.4.2"/' .mise.toml
mise install

mise exec -- bun --version now answers 1.4.2, and celilo --version starts. That method was not approved. It patches the incident rather than the tool, and celilo has no record of it. The operator's instruction was to file the general problem rather than keep doing this. The hand edit is still in place (backup: /var/celilo/.mise.toml.bak-20260917). The event dispatcher started before the edit and stays on 1.3.3 until it restarts.

Acceptance

An ordinary celilo apt-upgrade leaves the management server on the bun the installed celilo release pins, with no SSH and no hand edit.
Operator edits are still respected, detected properly: for example, record the checksum of the template each install wrote, and replace the file only if it still matches that checksum. That is the dpkg conffile rule, applied by hand. Alternatively, move the bun pin out of the operator-editable file altogether.
A changed pin takes effect everywhere celilo runs, including a dispatcher already running, rather than only for new invocations.
celilo system audit reports it when the server's bun differs from the pin the installed release shipped.
Recurrence gate: a test that installs a deb pinning bun A, upgrades to a deb pinning bun B, and asserts the server ends up on B. It should also assert a genuinely operator-modified pin survives the upgrade.

#1367: the merge gate running bun 1.3.3 against a pinned 1.4.2, the same drift on a different host.
#1376: oven/bun:latest unpinned in the e2e management image, the same drift one layer down.
#1378: no celilo route to reboot the management server. A reboot is also what finally moves the dispatcher onto a changed pin.

The celilo deb pins bun through `/var/celilo/.mise.toml`, but it writes that file **only on first install**. Every later upgrade leaves the pin alone, so the management server runs whatever bun it was first installed with, forever. There is no celilo command to change it either. The only way to move it today is to SSH in and edit the file by hand. ## Measured on celilo-mgr, 2026-09-17 ``` /var/celilo/.mise.toml bun = "1.3.3" written 2026-06-06 (first install) /etc/celilo/mise.toml.template bun = "1.4.2" shipped by the 3.0.0 deb mise exec -- bun --version 1.3.3 ``` `packaging/celilo/scripts/postinst:39-43`: ```sh # 3. Seed the conffile (does not clobber operator changes). if [ ! -f "$CELILO_HOME/.mise.toml" ]; then install -o "$CELILO_USER" -g "$CELILO_GROUP" -m 0644 \ /etc/celilo/mise.toml.template "$CELILO_HOME/.mise.toml" fi ``` The template pinned 1.3.3 from `50fbd69c` (2026-05-30) until `85e5665c` moved it to 1.4.2 (2026-09-04). celilo-mgr predates the move, so upgrades through 3.0.0 all skipped the write. ## Why it matters **celilo 3.0.0 was built and tested against bun 1.4.2 and was running in production on 1.3.3.** These are not interchangeable: the city's own instructions record that 1.3.3's `Bun.YAML.parse` reads a bare `on:` key as `true`, and a merge-gate test failed on exactly that. The pin is meant to guarantee the server runs the runtime the release was tested on, and on a long-lived server it guarantees the opposite. The guard's intent — *"does not clobber operator changes"* — is reasonable. The implementation cannot tell an operator's change from an untouched file an older package wrote, so it treats every existing file as operator-owned. ## What was done by hand, and why that is not the fix On 2026-09-17 around 06:21Z, crew-builder repinned the server by hand over SSH: ```sh cp .mise.toml .mise.toml.bak-20260917 sed -i 's/^bun = "1.3.3"$/bun = "1.4.2"/' .mise.toml mise install ``` `mise exec -- bun --version` now answers 1.4.2, and `celilo --version` starts. **That method was not approved.** It patches the incident rather than the tool, and celilo has no record of it. The operator's instruction was to file the general problem rather than keep doing this. The hand edit is still in place (backup: `/var/celilo/.mise.toml.bak-20260917`). The event dispatcher started before the edit and stays on 1.3.3 until it restarts. ## Acceptance - [ ] An ordinary `celilo apt-upgrade` leaves the management server on the bun the installed celilo release pins, with no SSH and no hand edit. - [ ] Operator edits are still respected, detected properly: for example, record the checksum of the template each install wrote, and replace the file only if it still matches that checksum. That is the dpkg conffile rule, applied by hand. Alternatively, move the bun pin out of the operator-editable file altogether. - [ ] A changed pin takes effect everywhere celilo runs, including a dispatcher already running, rather than only for new invocations. - [ ] `celilo system audit` reports it when the server's bun differs from the pin the installed release shipped. - [ ] Recurrence gate: a test that installs a deb pinning bun A, upgrades to a deb pinning bun B, and asserts the server ends up on B. It should also assert a genuinely operator-modified pin survives the upgrade. ## Related - #1367: the merge gate running bun 1.3.3 against a pinned 1.4.2, the same drift on a different host. - #1376: `oven/bun:latest` unpinned in the e2e management image, the same drift one layer down. - #1378: no celilo route to reboot the management server. A reboot is also what finally moves the dispatcher onto a changed pin.

forgejo-admin added the

severity:high

category:bug

labels

2026-09-17 06:36:21 +00:00

forgejo-admin referenced this issue

2026-09-17 06:56:59 +00:00

Make celilo-mgmt converge on a host with a stale or pre-existing HashiCorp apt source: the key refresh runs after the apt cache update that the stale key breaks #1380

forgejo-admin commented

2026-09-17 07:05:11 +00:00

Author

Owner