# ADR 023: K3s for declarative homelab workloads

- HTML version: https://robbiepalmer.me/projects/homelab/adrs/023-k3s-declarative-workloads
- Project: Home Lab (https://robbiepalmer.me/projects/homelab.md)
- Status: Proposed
- Date: 2026-09-01

# Context

Most homelab applications already run from OCI images. Docker Compose wires
those images together on the Mac mini, while launchd restarts the Compose
groups after a reboot. The current operational description also has launchd
running a shell wrapper around the Ente CLI, exporting the photo library to
the attached 10 TB disk. NixOS declares the Asus GPU worker. This leaves two
different kinds of desired state in the repository:

* host state, including operating systems, drivers, disks, USB devices,
  Tailscale, and the container runtime;
* workload state, including images, environment, ports, storage, health
  checks, dependencies, and placement.

Kubernetes does not need to own the first layer to provide value in the
second. NixOS can prepare a Linux node. Colima can provide a Linux VM on the
Mac. K3s can then reconcile the workloads running across those nodes.

The fleet is heterogeneous by design:

| Host            | Hardware constraint                               | Workload consequence                             |
| --------------- | ------------------------------------------------- | ------------------------------------------------ |
| Mac mini        | 10 TB media and photo-backup disk, plus macOS VPN | Storage-bound pods must run in its Colima VM     |
| Raspberry Pi    | Canon printer over USB and an SD-card boot disk   | CUPS belongs on the Pi; etcd does not            |
| Asus desktop    | GTX 1050 with 2 GB VRAM and CUDA 12               | Compatible GPU jobs need an explicit label       |
| RTX 4080 laptop | Planned, fast, and expected to sleep              | Newer GPU jobs need wake and availability checks |

Kubernetes supports that layout. Node labels and affinity constrain a pod to a
host or capability. Device plugins advertise GPUs as resources. Local
persistent volumes carry node affinity so a pod follows the disk it needs.
The nodes do not have to be interchangeable.

Images remove most package work from a workload. They do not configure the
host kernel, NVIDIA driver, Colima VM, disk mount, USB permissions, Tailscale,
or Wake-on-LAN. [ADR 022](/projects/homelab/adrs/022-ansible-k3s-migration-bridge)
uses Ansible only where the host layer or migration needs it.

# Decision

Run a bounded K3s pilot as the intended workload control plane. Keep the ADR
Proposed until the pilot proves storage, networking, restart, rollback, and
GPU behaviour on the real machines.

Start with one K3s server in an isolated Colima profile on the Mac mini. Use it
only for workloads that can tolerate loss of the Kubernetes API during Mac
maintenance. Do not describe this as a highly available cluster.

Join the Asus desktop as an agent through its NixOS configuration after the
Mac pilot works. Add the 4080 laptop as another agent after choosing its
operating system and remote transport, then testing its wake method and CUDA
path. The Raspberry Pi may join later as an agent, but it will not run the K3s
server or etcd while it boots from an SD card.

Keep both AdGuard Home instances outside K3s during the pilot. Their current
independence matters more than using one workload API everywhere.

```mermaid
flowchart TB
Git["Git repository"]
HostConfig["Host configuration<br/>NixOS and Ansible bridge"]
Manifests["K3s manifests<br/>workload state"]
Apply["mise apply and verify"]

subgraph mac["Mac mini"]
MacDNS["AdGuard Home<br/>primary, outside K3s"]
EnteExport["Ente CLI export<br/>launchd during pilot"]
PhotoDisk[("Photo export<br/>10 TB disk")]
Colima["Colima VM"]
Server["K3s server<br/>single control plane"]
Media["Media and knowledge workloads"]
end

subgraph pi["Raspberry Pi"]
PiDNS["AdGuard Home<br/>replica, outside K3s"]
CUPS["CUPS<br/>outside pilot"]
PiAgent["Optional K3s agent"]
end

subgraph asus["Asus desktop"]
NixOS["NixOS"]
AsusAgent["K3s agent<br/>GTX 1050"]
end

Laptop["Planned K3s agent<br/>RTX 4080"]

Git --> HostConfig
Git --> Manifests --> Apply --> Server
HostConfig --> Colima --> Server --> Media
HostConfig --> EnteExport
EnteExport --> PhotoDisk
HostConfig --> NixOS --> AsusAgent
Server --> AsusAgent
Server -.-> PiAgent
Server -.-> Laptop
MacDNS ~~~ Server
PiDNS ~~~ PiAgent
CUPS ~~~ PiAgent

classDef source fill:#4338ca,color:#fff,stroke:#3730a3
classDef cluster fill:#2563eb,color:#fff,stroke:#1d4ed8
classDef native fill:#0f766e,color:#fff,stroke:#115e59
classDef proposed fill:#fef3c7,color:#78350f,stroke:#d97706,stroke-dasharray:5 5
class Git,HostConfig,Manifests,Apply source
class Colima,Server,Media,NixOS,AsusAgent cluster
class MacDNS,EnteExport,PhotoDisk,PiDNS,CUPS native
class PiAgent,Laptop proposed
```

## What K3s owns

K3s will own these workload concerns:

* image versions and pull policy;
* Deployments, StatefulSets, Jobs, Services, and health probes;
* ConfigMaps and references to secrets;
* local persistent-volume claims and node affinity;
* node labels, taints, tolerations, and GPU resource requests;
* application provisioning Jobs built from the existing Python clients;
* restart and reconciliation after a container or pod failure;
* workload status exposed through mise commands.

Use checked-in manifests with Kustomize overlays where hosts need different
values. Run apply, diff, status, logs, and rollback commands through mise.
K3s will reconcile objects after they reach its API. The pilot does not need a
GitOps controller.

K3s can auto-apply files from
`/var/lib/rancher/k3s/server/manifests`, but deleting a file does not delete
the corresponding object. In a multi-server cluster, another process must keep
those files in sync. Use explicit repository apply and delete commands for
application workloads during the pilot.

## What remains host state

K3s will not own:

* macOS, Homebrew, Colima, launchd, or the host VPN;
* Linux boot configuration, kernel modules, firewalls, Tailscale, or SSH;
* NVIDIA kernel drivers and the container toolkit;
* mounting the Mac media disk into the Colima VM;
* exposing the Pi's printer USB device;
* Wake-on-LAN and laptop power policy;
* K3s installation, node tokens, or control-plane backup;
* the first delivery of a secret to the cluster.

NixOS owns those concerns on the Asus desktop. ADR 022 may bridge them on the
Mac and Pi until a nearer declarative owner takes over.

## Placement on a heterogeneous fleet

Use labels for capabilities rather than hard-coding node names into every
pod. The first label set should cover:

```text
homelab.dev/storage-media=true
homelab.dev/storage-photo-backup=true
homelab.dev/device=printer
homelab.dev/gpu=gtx-1050
homelab.dev/gpu=rtx-4080
homelab.dev/cuda=12
homelab.dev/power=on-demand
```

A media workload requests `homelab.dev/storage-media=true`. A CUDA 12 job can
request the GTX 1050 or another compatible worker. Newer jobs can require the
4080\. The NVIDIA device plugin exposes the GPU resource separately from these
compatibility labels.

Use separate local persistent volumes for the media library, Ente export, and
any node-owned DVC cache, even when two volumes currently share one physical
disk. Local storage remains unavailable when its node is down. Kubernetes
records that fact and keeps the pod on the correct node; it does not turn one
disk into shared storage.

Images that may run on both the ARM Pi and x86 workers need a multi-platform
manifest. Otherwise, constrain the pod to a compatible architecture.

## Workload migration

| Workload                                             | Pilot treatment           | Reason                                                                |
| ---------------------------------------------------- | ------------------------- | --------------------------------------------------------------------- |
| SilverBullet                                         | Move first                | Small Compose stack with simple storage and rollback                  |
| Jellyfin                                             | Move after SilverBullet   | Exercises the media volume, LAN playback, and restart recovery        |
| Prowlarr, Sonarr, Radarr, qBittorrent, and Recyclarr | Move as one bounded stage | Shares storage and the macOS VPN dependency                           |
| Jellyfin and Servarr provisioners                    | Package as Jobs           | Keeps the existing API reconciliation code                            |
| Ente CLI photo export                                | Keep in launchd first     | Protects a core backup; reassess as a CronJob after storage tests     |
| GPU batch work                                       | Add after the Asus joins  | Proves labels, the device plugin, and CUDA 12 images                  |
| CUPS                                                 | Defer                     | Needs Pi USB access and a privileged device boundary                  |
| Netdata                                              | Split later               | A pod sees cluster and VM state; a host agent sees macOS and Pi state |
| AdGuard Home pair                                    | Keep outside              | Preserves independent DNS recovery during control-plane outages       |

Keep each Compose definition until its K3s replacement passes two unchanged
applies, a host reboot, an application rollback, and a storage recovery test.
Do not migrate the whole Mac in one change.

## Ente photo export

The Ente export counts as a workload even though it currently runs outside a
container. It has a command, schedule, credentials, destination volume,
success state, and failure policy. A Kubernetes CronJob could declare those
parts and keep Job history. [Ente's CLI documentation](https://ente.com/help/photos/faq/advanced-features#how-do-i-use-the-cli-to-export-my-photos)
describes incremental exports which tolerate interruption, so the workload
shape fits a CronJob.

Keep the current launchd path during the first K3s pilot. This export provides
the local recovery copy of the photo library. Moving it would add Colima, K3s,
the control plane, an image, and the VM-to-host mount to its failure path. The
repository does not yet contain the wrapper or plist, so ADR 022 first captures
the working deployment and makes it reproducible.

Reassess it after local-volume recovery and scheduled Job monitoring work on
the Mac. A K3s replacement must:

* build and pin an image from Ente's CLI Dockerfile rather than using the
  Ente server image as an export client;
* mount only the photo-export volume and select the Mac through
  `homelab.dev/storage-photo-backup=true`;
* read Ente configuration and credentials from a Secret-backed mount without
  exposing them in manifests, events, or logs;
* set `concurrencyPolicy: Forbid` and fail when the destination volume is
  absent, full, or read-only;
* report the last successful export and alert on failure or staleness;
* keep the photo tree outside agent and unrelated workload permission scopes;
* prove two successful scheduled exports and a restore inspection before
  disabling launchd.

Never run two schedulers against the same export. Keep the launchd files as the
rollback path until the CronJob has passed those checks.

## The Mac boundary

Colima supports K3s inside its Linux VM, but the useful workloads depend on
resources outside that VM. The pilot must prove each boundary on the actual
Mac:

* the VM starts at boot without an interactive session;
* the 10 TB volume mounts before storage-dependent pods start;
* LAN and tailnet clients can reach the required Services;
* UDP and TCP exposure work where a workload needs both;
* media traffic cannot bypass the macOS VPN gate;
* a Colima or macOS upgrade does not silently replace cluster state;
* a fresh VM can restore the K3s datastore and manifests.

The current `keep-running.sh` VPN check cannot simply disappear. Replace it
only after a tested mechanism prevents qBittorrent and its peers from starting
or reaching the internet without the required tunnel.

## Control-plane limits

The first cluster has one server and an embedded SQLite datastore. If the Mac
or its Colima VM is unavailable, the Kubernetes API and reconciliation stop.
SilverBullet and a reversible media pilot can accept that outage. Household
DNS cannot accept it as the only recovery path.

Embedded-etcd high availability needs at least three server nodes and an odd
number for quorum. The current fleet does not have three suitable members:

* The Raspberry Pi boots from an SD card. K3s warns that etcd is write-heavy
  and recommends an external SSD for Pi servers.
* The Asus desktop currently uses a tethered network path and cannot always
  recover after sleep.
* The plan puts the 4080 laptop to sleep when idle.

Do not solve that by labelling unreliable machines as control-plane servers.
Revisit high availability after the lab has three durable, independently
recoverable server nodes or an external datastore with a tested recovery
plan.

## Secrets

A Kubernetes Secret is an API object, not an encrypted source of truth. Do not
commit plain credentials or generated Secret manifests.

The pilot may create Secrets from the current gitignored local files through a
mise command that does not print their values. Choose repository-backed secret
encryption in a separate decision before adding unattended GitOps. The choice
must explain key recovery from a new operator machine.

## Pilot plan

1. Create an isolated Colima profile with pinned resources and K3s enabled.
   Add backup, restore, apply, diff, status, and logs commands to mise.
2. Deploy SilverBullet with a local persistent volume and a different test
   port. Verify data survives pod deletion, K3s restart, Colima restart, and a
   Mac reboot.
3. Replace the Compose SilverBullet path only after the K3s path passes and
   rollback to Compose works.
4. Test the Mac storage mount, a disposable CronJob, LAN exposure, Tailscale
   access, and the VPN gate without using the real photo export. Then migrate
   Jellyfin and the media group in separate changes.
5. Reassess the Ente exporter after storage recovery and Job monitoring have
   passed. Keep launchd if K3s makes the backup path less reliable; otherwise,
   migrate it under the safeguards above.
6. Enable a K3s agent in the Asus NixOS configuration. Install the NVIDIA
   device plugin and run the existing CUDA 12 `nvidia-smi` test as a Job.
7. Add the 4080 laptop after commissioning. Prove wake, job placement, and
   idle shutdown without making the cluster depend on the laptop.
8. Reassess the Pi after the Mac and Asus paths work. Keep it out of etcd
   unless its boot and data storage move to a suitable SSD.

## Pilot acceptance criteria

Change the status to Accepted only after the pilot proves these points:

* Checked-in manifests converge to zero changes on a second apply.
* SilverBullet data survives pod, K3s, Colima, and Mac restarts.
* Removing the K3s workload and restoring the Compose version has a documented
  and tested path.
* The media disk cannot bind to a pod on the wrong node.
* LAN and tailnet clients reach migrated services without public exposure.
* The VPN-gated stack cannot send traffic when the host tunnel is down.
* The Ente export has one active scheduler and fails closed when its
  destination is not mounted. If it moves to K3s, two scheduled exports and a
  restore inspection must pass before disabling launchd.
* Losing the K3s server does not stop either native AdGuard resolver.
* A datastore backup restores the pilot into a fresh Colima profile.
* The Asus joins through its NixOS configuration and completes a CUDA 12 GPU
  Job.
* Application provisioning Jobs remain idempotent.
* No manifest, event, normal log, or verbose command output exposes a secret.
* The runbook states which services remain outside K3s and why.

The 4080 laptop and Pi agent are not acceptance dependencies. They must follow
the placement and recovery rules above when they join.

# Alternatives considered

## Keep Docker Compose and launchd

Compose works well for one engine and has little overhead. launchd already
recovers the Mac stacks after a reboot.

The model stops at the Mac VM. It cannot declare GPU Jobs on the Asus worker,
place work on the planned laptop, or use one API for workload status across
the fleet. Keep Compose as the rollback path during the pilot, not the target
workload API.

## Use full Kubernetes

Kubeadm exposes every control-plane choice. This lab does not need that
assembly work. K3s packages the API server, controller, scheduler, container
runtime, networking, and common add-ons for small Linux systems while keeping
the Kubernetes API.

Choose K3s unless the pilot finds a concrete incompatibility.

## Use Nomad

Nomad has a smaller workload model and handles mixed clients well. The current
images could run there.

Kubernetes has the broader ecosystem for GPU device plugins, storage objects,
health probes, Jobs, and declarative application wiring. Colima also supports
K3s directly. Prefer the path already supported by the Mac runtime.

## Move every service at once

This would reach the target faster on paper. It would also combine storage,
VPN, DNS, USB, GPU, and control-plane failures into one rollback.

Move low-risk workloads first. Keep DNS independent until the cluster has an
honest availability story.

# Consequences

## Positive

* One API describes container workloads across the Mac and GPU workers.
* Images, manifests, health checks, placement, and storage claims live in Git.
* Fixed hardware placement becomes explicit instead of hidden in directory
  names and shell commands.
* Existing Python reconcilers can run as Jobs instead of becoming YAML.
* The 4080 laptop joins an existing worker model after commissioning.
* Ansible remains a removable bridge rather than another workload platform.

## Negative

* K3s adds a control plane, containerd, CNI networking, and Kubernetes object
  semantics to a small lab.
* The first server is not highly available.
* Local volumes preserve placement but cannot move data after a node failure.
* macOS resources remain behind the Colima VM boundary.
* Debugging now crosses host, VM, cluster, and application layers.
* The migration temporarily keeps Compose and K3s definitions side by side.

## Revisit triggers

* Stop the migration if SilverBullet cannot survive routine Colima and Mac
  restarts without manual repair.
* Reconsider the single-server design before moving a service whose recovery
  matters during Mac maintenance.
* Reconsider K3s if the VPN boundary or media storage becomes less reliable
  than the current Compose path.
* Keep the Ente exporter under launchd if the CronJob adds dependencies
  without improving recovery, observability, or control of overlapping runs.
* Revisit embedded-etcd high availability only after three suitable server
  nodes exist. Moving the Pi to an SSD would remove one blocker, not every
  blocker.
* Add GitOps only when unattended reconciliation is worth another controller
  and the secret-encryption design is ready.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt