Engineering

The Ubuntu Server That Did Not Need Nix

September 7, 202612 min read

The public VPS had eleven newer kernels installed and was still running the one from May. I had arrived to migrate Ubuntu to NixOS. Within minutes, the honest answer was no.

Part 3 of Three Machines, One Flake, a migration across a laptop, a home server, and a public VPS.

TL;DR

Infrastructure work should target the active boundary, both when choosing a change and when verifying it. On this VPS, the decision not to migrate Ubuntu to NixOS meant activating the installed kernel, reducing public access, bounding logs, recovering configuration, and owning failed automation before considering an operating-system migration.

It was a public Ubuntu 24.04 VPS with two cores, 3.7 GB of RAM, and twenty-two weeks of uptime. Caddy sat in front of five Docker containers: two applications, Postgres, and two monitoring agents. Compose projects lived under /root/press and four directories in /opt. Sixteen root cron jobs drove publishing, SEO work, social synchronization, and backups.

Nix could manage the shell. NixOS could manage the machine after reinstalling the operating system underneath a live public server. Neither option addressed the active risks.

Installed updates were not running

The box had been receiving unattended upgrades for five months:

$ uname -r
6.8.0-90-generic
 
$ cat /var/run/reboot-required.pkgs | tr ' ' '\n' | sort -u | head -4
libc6
linux-image-6.8.0-106-generic
...
linux-image-6.8.0-138-generic

Eleven kernels and two libc updates were installed. The machine was still executing the May kernel. Package management considered it patched; the process table did not.

The missing operation was a scheduled reboot. I managed the interruption explicitly, then verified the running kernel after roughly thirty seconds. That removed an immediate risk without committing the server to days of migration work.

The same audit found password authentication enabled on a public SSH port and 7,040 failed logins in twenty-four hours. Root already accepted keys only and there were no other user accounts, so password authentication protected nothing. Disabling it removed disk, CPU, and noise from a path nobody needed. Because the server was already reachable through a private network, the stronger follow-up was to remove public SSH exposure rather than add fail2ban. That follows the same access pattern described in How I reach a home server with no public address.

Docker cleanup reclaimed 16.5 GB and moved disk usage from 65% to 38%. These outcomes did not require a new package manager.

A backup that would break the service

/etc/caddy contained four hand-created Caddyfile backups. The newest predated a routing change. Restoring the “latest” safety copy would have taken a site offline.

I committed the four historical files first so their states remained recoverable, then removed the ad hoc copies and put the live configuration under version control.

There was a related temptation under /opt: run git init across the directory and capture the remaining compose files. Inspection found five .env files plus credentials.json and credentials-token.json. A blanket repository would have staged secrets alongside infrastructure.

The repository therefore starts with a deny-by-default ignore file:

*
!*/
!docker-compose.yml
!compose.yml
!Caddyfile

The allowlist can grow from there. git check-ignore -v must prove every credential remains excluded before the first commit. Version control is useful only after deciding what it must never see.

Cloudflare was optional to the origin

TLS terminated at Cloudflare. The origin Caddyfile listened on plain port 80 with a catch-all handler, and UFW allowed the port from anywhere.

Every Cloudflare control, including WAF, rate limits, and bot filtering, could therefore be bypassed by addressing the origin directly:

$ export ORIGIN_IP="203.0.113.10"
$ curl -sD- -o /dev/null \
    -H 'Host: studio.example.dev' "http://$ORIGIN_IP/"
HTTP/1.1 307 Temporary Redirect
Location: /login
Server: Caddy
 
$ curl -s -o /dev/null -w '%{http_code}\n' \
    -H 'Host: nonsense.invalid' "http://$ORIGIN_IP/"
302

There was no cf-ray because Cloudflare never saw the requests. Even an unknown host reached an application.

The application behind that route made the bypass consequential. Studio compared a static password with === and issued a session cookie containing an HMAC of a fixed message. It had no rate limit or lockout. The token never expired and could only be revoked by rotating its secret. The container ran as root and carried roughly twenty live credentials in its environment.

Nix would not have changed this boundary.

The networking remediation has two defensible forms:

allow only Cloudflare’s published origin ranges and return 403 for unmatched hosts; or
use a Cloudflare Tunnel and remove the public origin listener entirely.

The allowlist preserves the current architecture but requires maintenance when Cloudflare’s published ranges change. A tunnel removes the listener while adding another control plane. The audit identified that tradeoff rather than pretending package management could decide it. Until one path is implemented and tested, the origin remains an open remediation item. How to Harden Cloudflare WAF Without Bot Management covers the adjacent edge-security boundary.

Two crons failed 720 times a day

Two of the sixteen root cron jobs ran every two minutes. A laptop-side package-manager change removed the mise shim path one of them expected over Tailscale.

The server recorded the consequences in Postgres:

2026-09-06 | 177  # every run so far
2026-09-05 | 720  # every run that day
2026-09-04 | 671
2026-09-03 | 128  # normal laptop-asleep baseline
2026-09-02 |   5

Fifteen thousand five hundred failure rows accumulated, 46% of the audit table and 15 MB of a 72 MB database. Each failed run launched docker exec ... psql to record its failure.

The cause lived on another machine, in another repository, behind an untracked script. Nothing on the VPS escalated it. The local Gatus instance monitored one endpoint on a different machine while ignoring Caddy, both applications, and Postgres beside it.

Part 2 had already shown that the alerting stack detected the dead backup correctly. Adding a check was insufficient because a signal without an owned response path is only a better-organized log.

Unbounded writers

Three writers could grow until the disk forced the issue. /var/log/press-social.log had reached 12 MB and 187,000 lines without rotation since June. The systemd journal occupied 2.1 GB and was heading toward its default percentage-based ceiling. Docker used json-file logs without a size limit and outside the existing logrotate stanza.

One monitoring container had accumulated 69 MB at roughly 1.35 MB per day. Recreation resets container logs, disguising growth during active deployment periods.

btmp.1 had reached 126 MB and auth.log.1 23 MB for one day. Those were symptoms of the 7,040 failed SSH attempts, not separate retention problems. Key-only root access made the attempts noise rather than an authentication path, but leaving an unnecessary public interface open still imposed a cost.

A container had lost its compose file

docker compose ls reported a project whose working directory contained only data/. The compose file describing the running container had moved.

The container survived because of restart: unless-stopped, so reboots worked while management did not. Running down from the compose file that remained one directory away would not stop the orphan. Starting it could collide on container_name.

Runtime state had remained healthy after the artifact that owned it disappeared.

When not to migrate Ubuntu to NixOS

NixOS would have made service definitions declarative and upgrades atomic, with configuration reviewable in pull requests. Those are real benefits.

It would not have activated the installed kernel, closed password authentication, bounded the logs, put Caddy under version control, detected the broken cron response, moved backups off the same physical disk, or removed the public origin path.

Reinstalling a live 3.7 GB server would have consumed the attention those fixes required. It would have solved the problem I knew how to solve while the May kernel continued running.

Two of three machines got Nix because their problems matched what Nix addressed. The third got a verified reboot, Docker cleanup, and a small Git repository. Network exposure and log bounds remained explicit remediation work rather than disappearing inside a migration plan.

Six checks passed beside the target

After the three machine passes, four independent reviews ran in parallel: one per host and one across the repository. Most serious findings had been introduced by the migration itself. The checks that might have caught them observed nearby state instead of the active boundary.

Check	What it appeared to prove	What it measured
`nix eval .#darwinConfigurations.laptop.config.launchd.user.agents`	no mise path remained	the attribute query failed; `grep -c` returned zero on empty output
`shellcheck scripts/*.sh \| tail; echo $?`	CI was fixed	`tail`’s exit code; CI remained red on six of six runs
`nix flake check`	linked files existed	`home.file.<name>.source` was never forced
`packages.lock`	the package set could not shrink	only `home.packages`; not the Homebrew list that `cleanup = "zap"` deletes
`mise current`	project pins resolved	what configuration declared, not what the shell executed
`drift-check`	the machine matched Git	a `find` expression whose `-prune` never ran; it wedged for 2h19m

The first check compared evaluated configuration with a rendered plist, although launchd loads the rendered file. The home.file assertion also remained inert because evaluation never forced .source.

The pipeline reported tail’s status instead of shellcheck’s. A separate version comparison consulted a registry while ignoring the locked flake that supplied the installed versions.

Mise reported its declared selection without proving which binary the shell resolved. The package lock covered the reversible, generation-managed set but omitted the Homebrew casks that cleanup = "zap" could delete.

Each check returned success without a warning or diff. In these cases, green meant either “correct” or “the target was never observed.” That ambiguity made the checks dangerous because it retired the suspicion that could have exposed the failures.

Test the guard against the failure

Two stronger guards were written after something escaped.

The first makes an out-of-store link assert that its repository target exists:

link = path:
  assert lib.pathExists (../../home + "/${path}")
    || throw "home/${path} does not exist";
  config.lib.file.mkOutOfStoreSymlink "${repo}/${path}";

An assertion is still inert unless evaluation forces it. A dedicated flake check walks the home.file values for every host and forces each .source.

Then I tested the check against its actual threat:

$ git mv home/.config/starship.toml \
    home/.config/starship-RENAMED.toml
$ nix flake check --no-build
error: home/.config/starship.toml does not exist
$ echo $?
1

Before that experiment, the same rename returned zero.

The package lock received the same treatment: remove a package or cask and confirm the lock diff appears. The useful lock covers the list that zap can delete, where a false negative has destructive consequences.

The longest silent regression

Twenty-two .pre-nix files remained after adoption. They were Home Manager collision backups waiting to be deleted. Diffing each against its live counterpart uncovered six settings missing from ~/.claude/settings.json:

hooks                   PreToolUse -> Bash -> "rtk hook claude"
permissions             defaultMode: auto
env                     PILOT_* review flags
model                   opus[1m]
modelSettings           effortLevel: medium
extraKnownMarketplaces  two plugin sources

The tracked file had never contained them. They had accumulated locally over months and existed only in the live copy. Home Manager linked the repository version over it, and a token-saving pre-tool hook stopped running for three days without an error.

backupFileExtension preserved the old file, making recovery possible while leaving the loss silent. Once adoption finished, I removed the setting so future collisions stop activation instead of moving the existing file aside. The twenty-two stale backups also carried a latent failure: Home Manager refuses to overwrite an existing backup, so an unrelated future switch could have aborted.

Why fresh review found the failures

Each reviewer received the established context and deliberate decisions, plus one instruction: prefer five evidenced findings over twenty speculative ones, and verify before asserting.

One reviewer challenged my claim that the monitoring stack had no rules. It first checked whether the queried resource types existed, then found the opposite conclusion in vmalert’s API. The repository reviewer opened Actions and found the permanently red workflow. I could have found it during the three-day migration, but my local pipeline had reported the wrong process’s status and convinced me to stop looking.

Fresh attention helped because my own next pass would have inherited the assumptions behind the first explanation.

What remains

The migration was worth doing. Two Macs now rebuild from one command. Package and configuration drift are visible. Thirty-two commits replaced four uncoordinated layers with twenty-seven files. On the public server, the running kernel was brought current, unused Docker data was removed, and live Caddy configuration gained versioned ownership. Origin restriction, public SSH removal, log bounds, backup placement, and the cron response path still require explicit implementation and verification.

The broader lesson is about infrastructure boundaries. A proposed change should address the risk that is active now, and its guard should inspect the artifact the system actually uses. On these machines, that meant checking the loaded plist, shell resolution, forced link sources, backup age, and the relevant process exit status. Failure injection then showed whether each guard could observe the condition it claimed to protect.

The closing figures were checked against the live machines: 149,888 packages in the compared nixpkgs indexes, 51,824 shell-history commands, nix-darwin module source at HEAD, and defaults read snapshots captured before and after each switch. Version numbers will drift. The work made that drift visible.

FAQ

Should a production Ubuntu server be migrated to NixOS?

Only when the operating-system migration addresses the highest active risks and the recovery plan justifies the interruption. On this VPS, rebooting into installed security updates, restricting network exposure, and recovering configuration ownership delivered more value with less risk.

Why are installed kernel updates not enough?

The package database can show a new kernel while the machine continues running the old one. Compare uname -r with the installed packages and /var/run/reboot-required.pkgs, then schedule and verify the reboot.

How do you test that an infrastructure guard works?

Trigger the failure the guard claims to catch and require a non-zero result. Renaming a managed target, removing a locked package, or breaking a command in a pipeline proves whether the check observes the real boundary.

What makes a green infrastructure check misleading?

It may measure the wrong target, read the wrong process’s exit code, or ask a configuration tool what it declared instead of inspecting the runtime artifact. A useful check must fail when its protected condition is deliberately violated.