# ADR 010: NixOS GPU worker

- HTML version: https://robbiepalmer.me/projects/homelab/adrs/010-nixos-gpu-worker
- Project: Home Lab (https://robbiepalmer.me/projects/homelab.md)
- Status: Proposed
- Date: 2026-08-02

# Context

I have an Asus desktop with a GTX 1060 — Pascal-era (`sm_61`), released 2016,
and now "unsupported" by the modern toolchain: CUDA 13 and the 585+ driver
branches dropped Pascal entirely. Rather than e-waste it, I'm turning it into
a headless, wake-on-demand GPU worker for batch computer vision jobs — the
kind of workload that needs many GPU-hours, not a modern GPU.

Managing dependencies on end-of-life hardware is fiddly: the driver, kernel,
and CUDA versions all have to be pinned just so, and there's no vendor telling
you what still works. That's exactly the kind of detail I want declared in a
config file, not remembered. It's also the kind of system my coding agents
can maintain well: the whole config is a git repo, so every change is
reviewable and rollbackable — the same config in produces the same system
out. An agent can edit a flake and I can review the diff, instead of someone
SSH-ing in and hand-editing driver settings on a box that then forgets them.

Nothing is built yet. If this is accepted, the host config for `asus-desktop`
will live in a `homelab/` directory in this repo.

# Decision (proposed)

Set up the desktop as a **NixOS** configuration managed with flakes,
declared in this repository, with the GPU stack deliberately pinned:

* An **LTS kernel** (6.12) plus the **580 driver branch** — the last with
  Pascal support — on `nixos-25.11`, locked there on purpose.
* **NVIDIA container toolkit + Docker**, so models run via
  `nvidia/cuda:12.x` images only (`--gpus all`). CUDA 13 dropped Pascal, so
  the CUDA 12 image family is the ceiling.
* **Tailscale** enabled, with the `tailscale0` interface trusted in the
  firewall and SSH for access from the Mac mini / laptop.
* **Wake-on-LAN** on the real interface plus BIOS power-on-by-PCI-E, so the
  box sleeps when idle and is woken on demand. The WoL magic packet is a
  Layer 2 broadcast that can't route over the tailnet — it's sent by the Mac
  mini on the home LAN, while the tailnet carries the wake request to it.
* A `robbie` user in the `docker` and `wheel` groups; no passwords committed
  to the repo.
* The **1TB HDD** reused as the data disk for models and datasets — Docker
  stays on the 111G SSD. It inherits the
  [DVC](/projects/homelab/adrs/008-dvc) approach as the local dataset remote.

# Alternatives

## Keep the old OS and install the driver by hand

* **Pros**: No migration effort.
* **Cons**: Non-reproducible; a kernel/driver update can silently break the
  GPU with no clean rollback — exactly the failure this project exists to
  prevent.
* **Decision**: Rejected. Declarative config is the point.

## Modern GPU / newer box instead

* **Pros**: No driver pins needed.
* **Cons**: Buys new hardware to do a job the existing box handles. The
  1060's batch-CV workload doesn't need modern hardware.
* **Decision**: Rejected. This is an ewaste-avoidance project.

## CUDA 13 / current drivers anyway

* **Pros**: Future-proof tooling.
* **Cons**: CUDA 13 and 585+ drivers dropped Pascal — the GPU simply won't
  work. Not a viable option, hence the pin.
* **Decision**: Rejected. The pin *is* the compatibility strategy.

# Consequences

### Pros

* **The 1060 keeps earning its place** doing batch CV work instead of going
  to landfill.
* **Reproducible and rollbackable**: a broken upgrade is a `git revert` and
  `nixos-rebuild switch` away, not a Saturday.
* **Upskilling in NixOS and heterogeneous orchestration** while fixing a
  real problem.
* **DVC integration**: the 1TB HDD becomes a local dataset remote shared
  with the ML pipelines.

### Cons

* **Frozen in time**: the box is pinned to nixos-25.11 and CUDA 12 — no new
  GPU features ever, and staying on the pinned channel is a deliberate,
  ongoing choice.
* **Docker-container-only GPU**: bare-metal CUDA 13 code won't run; must
  always run inside CUDA 12 images.
* **Wake-on-LAN discipline**: WOL only works if the BIOS, interface, and
  sleep config all cooperate; it's the most fragile part of the setup.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt