# ADR 010: NixOS GPU worker

- HTML version: https://robbiepalmer.me/projects/homelab/adrs/010-nixos-gpu-worker
- Project: Home Lab (https://robbiepalmer.me/projects/homelab.md)
- Status: Accepted
- Date: 2026-08-02

# Context

I have an Asus desktop with a GTX 1050 (2GB VRAM), Pascal-era (`sm_61`),
released 2016,
and now "unsupported" by the modern toolchain: CUDA 13 and the 585+ driver
branches dropped Pascal entirely. Rather than e-waste it, I'm turning it into
a headless GPU worker for batch computer vision jobs, the
kind of workload that needs many GPU-hours, not a modern GPU.

Managing dependencies on end-of-life hardware is fiddly: the driver, kernel,
and CUDA versions all have to be pinned just so, and there's no vendor telling
you what still works. That's exactly the kind of detail I want declared in a
config file, not remembered. It's also the kind of system my coding agents
can maintain well: the whole config is a git repo, so every change is
reviewable and rollbackable. The same config in produces the same system
out. An agent can edit a flake and I can review the diff, instead of someone
SSH-ing in and hand-editing driver settings on a box that then forgets them.

Accepted and deployed. The host config for `asus-desktop` lives in
`homelab/hosts/asus-desktop` in this repo, defined by a flake whose
`nixpkgs` input is locked to the `nixos-25.11` branch, and deployed with
`nixos-rebuild switch --flake`.

# Decision (proposed)

Set up the desktop as a **NixOS** configuration managed with flakes,
declared in this repository, with the GPU stack deliberately pinned:

* An **LTS kernel** (6.12) plus the **580 driver branch** (580.142, closed
  kernel modules; the open modules need Turing or newer), the last with
  Pascal support, on `nixos-25.11`, locked there on purpose.
* **NVIDIA container toolkit + Docker**, so models run via
  `nvidia/cuda:12.x` images only (`--device nvidia.com/gpu=all`, the CDI
  interface; the legacy `--gpus all` path needs a deprecated daemon setting
  and is not enabled). CUDA 13 dropped Pascal, so
  the CUDA 12 image family is the ceiling.
* **Tailscale** enabled, with the `tailscale0` interface trusted in the
  firewall and SSH for access from the Mac mini / laptop.
* **Wake-on-LAN** on the real interface plus BIOS power-on-by-PCI-E, so the
  box sleeps when idle and is woken on demand. Deferred for now: the desktop
  has no ethernet path to the router, and its uplink is USB tethering to an
  Honor Magic5 Pro, which drops in sleep and can't carry a magic packet. The WoL
  link policy is already in the config; waking waits on powerline ethernet
  adapters, and until then the box stays awake.
* A `robbie` user in the `docker` and `wheel` groups; no passwords committed
  to the repo.
* The **1TB HDD** reused as the data disk for models and datasets, mounted at
  `/data` (ext4, 916G free). Docker
  stays on the 111G SSD. It inherits the
  [DVC](/projects/homelab/adrs/008-dvc) approach as the local dataset remote.

# Alternatives

## Keep the old OS and install the driver by hand

* **Pros**: No migration effort.
* **Cons**: Non-reproducible; a kernel/driver update can silently break the
  GPU with no clean rollback. Exactly the failure this project exists to
  prevent.
* **Decision**: Rejected. Declarative config is the point.

## Modern GPU / newer box instead

* **Pros**: No driver pins needed.
* **Cons**: Buys new hardware to do a job the existing box handles. The
  1050's batch-CV workload doesn't need modern hardware.
* **Decision**: Rejected. This is an ewaste-avoidance project.

## CUDA 13 / current drivers anyway

* **Pros**: Future-proof tooling.
* **Cons**: CUDA 13 and 585+ drivers dropped Pascal. The GPU simply won't
  work. Not a viable option, hence the pin.
* **Decision**: Rejected. The pin *is* the compatibility strategy.

# Consequences

### Pros

* **The 1050 keeps earning its place** doing batch CV work instead of going
  to landfill.
* **Reproducible and rollbackable**: undo a broken upgrade with `git revert`
  and `nixos-rebuild switch`.
* **Upskilling in NixOS and heterogeneous orchestration** while fixing a
  real problem.
* **DVC integration**: the 1TB HDD becomes a local dataset remote shared
  with the ML pipelines.

### Cons

* **Frozen in time**: the box is pinned to nixos-25.11 and CUDA 12. No new
  GPU features ever, and staying on the pinned channel is a deliberate,
  ongoing choice.
* **2GB of VRAM**: batch sizes and model sizes are capped. Jobs must fit in
  memory or be split; the narrow-job framing matters more than it would on a
  bigger card.
* **Docker runs past the channel default**: the nixos-25.11 default Docker
  (28.5.2) is unmaintained upstream and vulnerable to CVE-2026-42306, a
  `docker cp` race fixed in 29.5.1. The config selects the same channel's
  maintained `docker_29` (29.6.0) explicitly, which clears the CVE without
  touching the nixpkgs pin.
* **Docker-container-only GPU**: bare-metal CUDA 13 code won't run; must
  always run inside CUDA 12 images.
* **Wake-on-LAN discipline**: WOL only works if the BIOS, interface, and
  sleep config all cooperate; it's the most fragile part of the setup.
* **Tethered uplink**: the phone is critical infrastructure. If USB
  tethering drops or the phone reboots, the desktop is unreachable until
  tethering is re-enabled.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt