The Home Server Where a Nix Migration Broke Two Jobs Silently

The migration preserved all 24 launchd property lists and killed the nightly backup for two days. The missing dependency was not in any plist: every job had quietly relied on a global mise shim path. The failure exposed a missing nix-darwin launchd PATH that existed only by accident.

Part 2 of Three Machines, One Flake, a migration across a laptop, a home server, and a public VPS.

This was a home server running an OrbStack Kubernetes cluster with 42 deployments, Flux GitOps, VictoriaMetrics, vmalert, Alertmanager, Grafana, roughly 27 self-hosted services, and 24 launchd jobs. Those jobs ran backups, reconciled a beancount ledger, refreshed a trading token, exported battery data, watched disks, and maintained services around the cluster.

It had no Nix, no managed dotfiles, and a 260-byte .zshrc.

The migration exposed four layers that I needed to verify separately:

1. Declared configuration: the Nix definitions I evaluated.
2. Rendered artifact: the plist files nix-darwin produced.
3. Runtime environment: paths, sockets, displays, credentials, and external applications available to each process.
4. Operational outcome: whether the job actually ran and produced current, usable work.

A migration can preserve one layer and fail at the next.

The plan also had to begin with the differences between hosts. Only thirteen tools were shared. Seven, age, fluxcd, gitleaks, restic, tmux, socat, and ffmpeg, were unique to the server. Forty-nine were unique to the laptop. Eleven of the thirteen shared tools were behind on the machine I touched less often.

That is what “latest” looks like without an owner.

Shared profile, separate machine

The flake settled into one shared tool profile and two host files. The laptop kept development tools, 21 casks, and Rectangle and Raycast preferences. The server kept four casks, the operations profile, and its agent definitions.

The separation was not cosmetic. A package list is the obvious host difference and usually the least dangerous one. Behavior hides in files that assume a physical environment.

Signing needed a person at the screen

The shared .gitconfig, for example, enabled commit signing and pointed gpg.ssh.program to 1Password’s op-ssh-sign. Applying it to a machine without 1Password made every commit fail. The fix was a shared identity plus a host include:

[include]
    path = ~/.config/git/host.conf

The laptop’s include enables signing. The server’s deliberately does not.

The same assumption existed one layer down. The shared .ssh/config pointed IdentityAgent at 1Password’s socket. On a host without that socket, outbound SSH failed with Permission denied (publickey). The migration also dropped OrbStack’s generated Include, which its own comment says must be the first line and which is how the local cluster resolves its SSH hosts.

Installing 1Password on the server did not solve the physical problem. The agent was enabled and the signing key fingerprint matched the laptop’s, but signing still prompted on a display nobody could approve while connected over SSH. Signing is disabled on that host with the reason written beside the setting. The declaration was valid, but its runtime assumption was not.

Adopting the launchd jobs

The launchd agents were the risky part. Transcribing them manually would have turned a migration into a reimplementation, so I dumped the existing property lists as JSON and generated the Nix definitions mechanically.

My counts came from different checkpoints. The initial inventory found 24 launchd jobs. The comparison and import batch contained 23: 18 timer-driven jobs and five keepalive services. The final system reported 25 declarative agents. My migration notes do not explain the one-job inventory gap or identify everything included in the final total, so those numbers should not be treated as a single clean before-and-after count.

Compare declared and rendered state

The first verification compared the evaluated configuration with the original plists. Nineteen of twenty-three were byte-identical. Four differed only because nix-darwin represented unset StartCalendarInterval fields as null values that disappear when the plist is written.

The rollout followed the cost of a mistake. I activated the eighteen timer-driven jobs first because a bad timer should surface in its own log on the next run. The five keepalive services followed because a bad definition there can become an outage or an infinite restart loop.

After activation, ~/Library/LaunchAgents still contained ordinary files:

$ ls -l ~/Library/LaunchAgents/ | head -3
-rw-r--r--  com.skippednote.battery.plist
-rw-r--r--  com.skippednote.disk-alert.plist
-rw-r--r--  com.skippednote.ledger.plist

That initially looked like a failed adoption. The ownership evidence lives elsewhere:

$ ls /run/current-system/user/Library/LaunchAgents/ | wc -l
18

Nix-darwin copies user-agent plists rather than linking them, and only replaces a live file when diff says the rendered result changed. A byte-equivalent transcription is supposed to be invisible in the user directory. I counted symlinks twice before checking the artifact that actually defines ownership.

Make the nix-darwin launchd PATH explicit

Removing Homebrew’s mise left 24 of 31 shims dangling. Two systems resolved tools through that directory, and both failed quietly.

The nightly backup died for two nights. Its complete log for each run was this:

03:30:05 ==== backup start 2026-09-06-033005 ====
03:30:05 ERROR: missing required command: restic

All eighteen imported agents had been copied from plists created when Homebrew and mise supplied their tools. None defined PATH. Launchd started them with /usr/bin:/bin, and the mise shim directory had been the accidental bridge to everything else.

The failure also crossed machines. A publishing cron on the public server dispatched work to the laptop over Tailscale through a script that hardcoded the mise shims directory in PATH. It ran every two minutes and failed 720 times a day for two days.

That rate accounts for 1,440 failed runs, not the 15,562 failure rows that made up 46% of the audit table. The larger figure therefore includes more than one row per scheduled failure or additional failures. The available audit summary does not distinguish them, so I cannot attribute all 15,562 rows to the two-day cron outage.

The migration preserved every plist field and still broke the jobs because the runtime dependency was absent from the plist. The nix-darwin launchd PATH had to become declared configuration instead of an interactive-shell side effect.

The fix applied an explicit environment to all agents while allowing specialized jobs to override it:

let
  agentPath =
      "/etc/profiles/per-user/skippednote/bin:"
    + "/run/current-system/sw/bin:"
    + "/Applications/OrbStack.app/Contents/MacOS/xbin:"
    + "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin";
 
  withPath = _: agent:
    lib.recursiveUpdate {
      serviceConfig.EnvironmentVariables = {
        PATH = agentPath;
        HOME = "/Users/skippednote";
      };
    } agent;
in {
  launchd.user.agents = lib.mapAttrs withPath {
    # the imported agents
  };
}

recursiveUpdate matters. The Hermes gateway pins Node through a store path and must keep its own PATH. A broad merge would have fixed eighteen jobs by breaking the nineteenth.

This fix makes the dependencies visible, but it does not make all of them reproducible. It still hardcodes the username, Homebrew’s mutable prefix, and OrbStack’s application path. Nix now owns the agent definition and part of its toolchain. Homebrew, OrbStack, and the host layout remain external runtime inputs that require separate verification.

Rendered files decide restarts

Before moving the keepalive services, I compared their evaluated definitions with the live plists and predicted that two would restart. All five reloaded.

Nix-darwin triggers from a diff of the rendered plist file, not the abstract configuration. Its serializer emits different ordering and whitespace from the handwritten XML. Two configurations can be semantically identical and still render different files, so my configuration-level comparison could not predict the file-level trigger.

All five services came back clean, so the bad prediction did not cause an outage. The method would have been expensive on a more sensitive service.

One keepalive failure had existed before Nix. Homebrew’s Atuin service was still registered after its formula disappeared:

$ plutil -extract ProgramArguments.0 raw -o - \
    ~/Library/LaunchAgents/homebrew.mxcl.atuin.plist
/opt/homebrew/opt/atuin/bin/atuin
 
$ launchctl list homebrew.mxcl.atuin
"LastExitStatus" = 19968

It was retrying a deleted binary forever. Because the daemon is an optimization rather than a dependency, the user-facing shell kept working and the restart loop could have survived for months. A Nix-declared sh.atuin.daemon replaced it with the store binary.

Checks at the outcome layer

The same four-layer model explained three findings outside launchd. I kept them in the audit because each one showed a valid local artifact producing an incomplete operational guarantee.

Cluster ownership

With 42 deployments, I expected some host schedules to duplicate Kubernetes CronJobs.

$ kubectl get cronjob -A
No resources found

The query showed that the resource type existed and had no instances. Most launchd work belonged on the host: ledger reconciliation, battery export, and OrbStack startup. Several jobs maintain cluster services from outside them. Moving those jobs into Kubernetes would change their failure domains rather than remove duplication.

Monitoring and freshness

I had reported that VictoriaMetrics, vmalert, Alertmanager, and Grafana were doing nothing because both of these returned empty:

$ kubectl get vmrule -A
$ kubectl get prometheusrule -A

They were empty because neither resource type existed. There was no operator. The alert rules were plain files mounted from a ConfigMap.

Vmalert’s own API showed 22 rules across seven groups. Every rule was healthy, four alerts were active, and all were routed to Telegram. The stack had correctly detected the dead backup for two days. Gatus had live probes. Detection succeeded, but nobody converted the alert into action. That gap is also why trusting a home server requires monitoring and backups, not merely valid configuration.

The restore check exposed a separate outcome gap:

$ cat ~/selfhost/logs/metrics/restore_check.prom
selfhost_restore_check_success 1
 
$ tail -1 ~/selfhost/logs/restore-check.log
restore-check OK /srv/backups/service-exports/2026-09-04-143239

The check ran today and verified a two-day-old export because the backup that should have produced a new one was dead. It measured whether verification completed, not whether the verified material was current. A backup checker that passes when there is nothing new to inspect validates its own schedule instead of backup freshness.

Remote Git state

The GitHub CLI token on the server was invalid. Neither local repository could fetch or push over HTTPS. ~/selfhost sat 110 commits ahead of its remote.

Flux used SSH and had already fetched the same commit, so the running cluster was not missing code. The drift checker’s “in sync with origin” step degraded to “could not fetch (offline?)”, and 110 commits of infrastructure history remained on one machine. The local repositories were valid, but the promised remote durability was missing.

What the home server gained

Homebrew fell from six formulae to one. Mise’s global list fell from fifteen tools to zero, and its install directory shrank from 2.3 GB to 570 MB. The server gained the same shell and package versions as the laptop without pretending the two machines had the same job. The final system reported 25 declarative launchd agents, subject to the accounting gap noted above.

The migration also broke two production jobs and produced an audit table with more than fifteen thousand failure records.

Launchd and Nix behaved as configured. I had treated a faithful transcription as a complete migration, even though the property lists omitted the package manager paths that made their commands resolvable. The reusable test is to follow each service from its declaration to the rendered file, then through its runtime environment to the work it produces. That same artifact-to-outcome distinction appears when verification schemas drift from runtime reality.

The durable lesson from the nix-darwin launchd PATH failure is simple: adopting a script means adopting its undeclared dependencies. Preserving its text preserves only the assumptions someone remembered to write down.

Next: the third machine is a public Ubuntu server. It needs a reboot, tighter networking, bounded logs, and recoverable configuration. Giving it Nix would delay those fixes.

FAQ

How does nix-darwin manage launchd agents?

It renders agent definitions into the current system and copies user plists into ~/Library/LaunchAgents. A regular file in that directory does not prove the agent is unmanaged; compare it with /run/current-system/user/Library/LaunchAgents and inspect the loaded service.

Why did the launchd jobs fail after the Nix migration?

The imported plists did not declare PATH. They had relied on Homebrew and mise shims from the interactive shell, so removing those shims left scheduled jobs unable to resolve tools such as restic.

How should launchd PATH be declared in nix-darwin?

Set serviceConfig.EnvironmentVariables.PATH for the agent collection, then use a recursive merge so specialised jobs can retain their own store-pinned path. Verify the job from launchd’s environment, not from an interactive shell.

Why can a backup check stay green when backups are stale?

A checker can successfully verify the newest artifact even when that artifact is days old. Its success condition must include freshness, such as the age of the latest completed backup, not only whether a restore command exited zero.