# ADR 022: Ansible as a bridge to K3s

- HTML version: https://robbiepalmer.me/projects/homelab/adrs/022-ansible-k3s-migration-bridge
- Project: Home Lab (https://robbiepalmer.me/projects/homelab.md)
- Status: Proposed
- Date: 2026-09-01

# Context

The homelab has three live computers and one planned GPU worker:

| Host            | State   | Job                                                                      |
| --------------- | ------- | ------------------------------------------------------------------------ |
| Mac mini        | Live    | Primary DNS, photo export, storage, media, monitoring, and coding agents |
| Raspberry Pi    | Live    | Secondary DNS, CUPS print server, and Netdata child                      |
| Asus desktop    | Live    | NixOS batch worker with a GTX 1050                                       |
| RTX 4080 laptop | Planned | Faster GPU worker for inference and training                             |

Configuration has grown around each service. NixOS declares the Asus desktop.
Docker Compose describes Mac container groups. Colima provides their Linux VM.
launchd and systemd start host services. Python programs configure Jellyfin
and the media applications through their APIs. The Raspberry Pi runs three
important services, but it has no host configuration under `homelab/hosts/`.

The Ente photo export is also missing from the repository. The current
operational description has launchd on the Mac mini running a shell wrapper
around the Ente CLI and writing to the attached 10 TB disk. Before changing
that path, inspect the host and confirm its label, schedule, script, CLI
version, configuration path, export destination, and failure reporting.

The repository contains twelve shell scripts with 1,230 lines and two Python
scripts with 597 lines. The Python contains application logic worth keeping.
The shell scripts repeat package installation, environment checks, launchd
rendering, Colima startup, Compose application, and health waits. They also
hide ordering. SilverBullet assumes Jellyfin has already configured the shared
Colima VM.

[ADR 023](/projects/homelab/adrs/023-k3s-declarative-workloads) proposes K3s
as the long-term owner of container workloads. That changes the job Ansible
needs to do. Rewriting every Compose bootstrap as an Ansible role would create
an intermediate workload system and then throw it away during the K3s move.

The machines need this work before they reach that target:

* The Mac needs reproducible Colima, K3s, storage-mount, boot, and scheduled
  Ente export setup.
* The Pi's current packages and service configuration need recording before
  any migration.
* The Asus worker needs a locked NixOS deployment that can join K3s without
  losing its CUDA 12 path.
* The fleet needs one inventory, ordered changes, and verification during the
  transition.
* The planned laptop will need commissioning after choosing its operating
  system and testing its wake behaviour.

# Decision

Pilot Ansible Core as a temporary bridge for host discovery, bootstrap,
rollout, and verification. Do not make it the workload control plane.

Ansible will run on demand from a trusted macOS or Linux operator machine.
The repository will hold its inventory, variables, playbooks, and roles. mise
will remain the command interface. Do not add AWX, Semaphore, Automation
Controller, or a permanent Ansible daemon.

The bridge has an exit rule. Every Ansible task must belong to one of these
categories:

1. Permanent host work that Kubernetes cannot perform, such as configuring
   Colima on macOS.
2. Deployment of another source of truth, such as applying the locked NixOS
   flake to the Asus desktop.
3. Temporary migration work with a named deletion condition.

Do not write an Ansible role for a container workload that ADR 023 assigns to
K3s. Kubernetes manifests will own those workloads directly.

## Ownership

| Concern                                                    | Owner                                                |
| ---------------------------------------------------------- | ---------------------------------------------------- |
| Host inventory and transition order                        | Ansible                                              |
| Operator commands                                          | mise                                                 |
| Asus operating system, driver, and container runtime       | NixOS flake                                          |
| Mac packages, Colima, host mounts, and launchd integration | Ansible during the pilot                             |
| Ente export schedule and wrapper while they remain native  | launchd, installed and checked by the Ansible bridge |
| Raspberry Pi service inventory and migration checks        | Ansible during the pilot                             |
| Container workloads, services, jobs, and placement         | K3s under ADR 023                                    |
| DNS filters and settings replicated to the Pi              | AdGuardHome-Sync                                     |
| Jellyfin and media application settings                    | Existing Python clients, later run as K3s Jobs       |
| Host process supervision                                   | launchd, systemd, and NixOS                          |
| Secrets                                                    | Gitignored local files or a future encrypted source  |

```mermaid
flowchart LR
Git["Git repository"]
Mise["mise"]
Ansible["Ansible<br/>migration bridge"]
Host["Host prerequisites<br/>inventory and checks"]
Nix["NixOS flake<br/>Asus host"]
K3s["K3s<br/>workload state"]

Git --> Mise
Mise --> Ansible
Ansible --> Host
Ansible -->|"deploy and verify"| Nix
Ansible -->|"bootstrap and verify"| K3s
Git -->|"manifests"| K3s

classDef source fill:#4338ca,color:#fff,stroke:#3730a3
classDef bridge fill:#b45309,color:#fff,stroke:#92400e
classDef target fill:#0f766e,color:#fff,stroke:#115e59
class Git,Mise source
class Ansible bridge
class Host,Nix,K3s target
```

## Inventory

Inventory the three live machines by tailnet name. Use groups for capabilities
and operating systems rather than one hierarchy that mixes both:

```yaml
all:
  children:
    always_on:
    dns_resolvers:
    compute_workers:
    macos_hosts:
    linux_hosts:
    nixos_hosts:
    k3s_servers:
    k3s_agents:
```

The Raspberry Pi can belong to `always_on`, `dns_resolvers`, and
`linux_hosts`. The Asus desktop can belong to `compute_workers`, `nixos_hosts`,
and later `k3s_agents`. Do not add the 4080 laptop to a live group until it has
a tailnet name and remote-management transport.

Use tailnet DNS names. Keep raw `100.x` addresses out of task files. Apply
host changes one at a time and stop after the first failure.

## Bridge responsibilities

The pilot may use Ansible to:

* gather facts and report missing prerequisites across the fleet;
* inventory the live Ente launchd job and export script without printing its
  credentials;
* record the Pi's package versions, service units, and non-secret
  configuration;
* install pinned Mac command-line tools;
* configure Colima's VM resources, K3s mode, storage mounts, networking, and
  launchd startup;
* deploy the exact NixOS Git revision to the Asus desktop;
* verify Docker or containerd, Tailscale, storage, service endpoints, and GPU
  access;
* configure a K3s agent on a non-Nix host if no nearer source of truth exists;
* wake and commission the 4080 laptop after its hardware path works;
* invoke the K3s apply and verification commands defined by ADR 023.

The Ente inventory must record the launchd label and schedule, Ente CLI
version and installation method, wrapper path, configuration path, export
destination, log path, monitoring check, and protection against overlapping
runs. Before starting an export, it must verify that the destination resolves
to the expected mounted volume. Separate secret-bearing values, then check the
wrapper and plist into the repository. Keep launchd as the active scheduler
during the first K3s pilot.

Prefer modules that read state before changing it. Commands need explicit
`changed_when` and `failed_when` rules. Support `--check --diff` where the
module can predict a change.

The pilot must not use Ansible to:

* restate NixOS packages, NVIDIA configuration, or systemd units as YAML;
* translate each Compose service into a long-lived Ansible role;
* supervise running applications;
* store plain secrets in inventory or role defaults;
* become a GPU job queue;
* install K3s on the Raspberry Pi as a server while it boots from an SD card.

## Protect the DNS pair

AdGuard Home remains outside K3s during the first workload pilot. The Mac and
Pi must keep recovering independently.

Ansible may verify the pair. If a later bridge task changes either resolver,
it must query both first, change the Pi replica, verify it from another host,
and stop before touching the Mac if that query fails. No handler may restart
both resolvers.

## Secrets

Keep the current gitignored environment files during the bridge. Any task that
handles a credential must suppress its output in normal and verbose runs.

[Ansible Vault](https://docs.ansible.com/projects/ansible/latest/vault_guide/index.html)
would encrypt values at rest but would not decide how a new control machine
gets the decryption key. Choose the long-term secret system with the K3s
deployment design. Do not let a temporary bridge choose it by accident.

## Migration plan

1. Add pinned Ansible tooling, inventory, a facts playbook, and read-only fleet
   verification.
2. Record the Pi's live services and the Mac's Ente export without changing
   them. Document the Pi's SD card, boot settings, packages, service units,
   CUPS queue, and AdGuard role. Record the Ente launchd job, wrapper, CLI,
   destination mount, logs, and alert.
3. Configure an isolated K3s pilot on the Mac through Colima. ADR 023 moves
   SilverBullet first and keeps its Compose rollback path.
4. Add the K3s agent and GPU verification to the Asus NixOS configuration.
   Ansible deploys and verifies the flake rather than configuring the agent
   itself.
5. Migrate Mac workloads according to ADR 023. Delete each replaced bootstrap
   path after two unchanged applies and a recovery test.
6. Decide whether the Pi should remain a host-service machine or join as a
   K3s agent. Do not make it an etcd member while it uses the SD card.
7. Remove bridge roles when NixOS, nix-darwin, Colima configuration, or K3s
   manifests take over their state.

## Pilot acceptance criteria

Keep this ADR Proposed until the bridge proves these points:

* Inventory reaches the Mac mini, Raspberry Pi, and Asus desktop by tailnet
  name and names each missing prerequisite.
* A read-only command reports the health of both DNS resolvers, CUPS, Netdata,
  the latest Ente export, other Mac workloads, and the Asus GPU path.
* The Ente wrapper and launchd definition are reproducible without committing
  credentials, and the job refuses to start when the 10 TB destination is not
  mounted.
* Ansible can reproduce the Mac prerequisites for the isolated K3s pilot.
* The Asus play deploys an exact flake revision and verifies a CUDA 12
  container without restating NixOS configuration.
* A second run against stable state reports zero changes.
* No K3s workload has two competing definitions in Ansible and Kubernetes.
* Each temporary role names the event that allows its deletion.
* The bridge removes at least one existing bootstrap script.
* Normal and verbose logs expose no secret.

# Alternatives considered

## Keep the shell scripts

This avoids a new tool. It also keeps the hidden ordering, inconsistent change
reporting, and missing Pi configuration. mise remains useful, but the scripts
do not provide a fleet inventory or common rollout guards.

Keep mise as the interface. Replace shell only where Ansible earns its keep.

## Make Ansible the permanent workload system

Ansible can render Compose files, start containers, wait for health, and call
application APIs. That would improve the current scripts.

ADR 023 assigns those jobs to K3s. Building permanent service roles first
would create two migrations and two sources of truth. Restrict Ansible to the
host and transition layers.

## Move every host to Nix first

NixOS already works well on the Asus desktop, and nix-darwin could reduce the
Mac role. Migrating the daily-use Mac and the working Pi before testing K3s
would expand the project and delay feedback on the workload design.

Use Nix where it already owns a host. Let the bridge shrink as later host
migrations prove worthwhile.

# Consequences

## Positive

* The fleet gets one inventory and one transition path.
* Ansible work stays small enough to delete.
* NixOS and K3s remain the long-term sources of truth.
* The Pi enters the recovery documentation before anyone changes it.
* The K3s pilot does not depend on a wholesale host migration.

## Negative

* The transition adds YAML, Jinja, and Ansible collection behaviour.
* Some host state will have both an old script and a bridge role until its
  migration finishes.
* A careless bridge task can still damage more than one host.
* Deleting temporary automation requires discipline after the migration.

## Revisit triggers

* Remove Ansible if NixOS, nix-darwin, and checked-in Colima configuration can
  reproduce every host prerequisite.
* Keep a small permanent Ansible layer only for host state with no better
  declarative owner.
* Do not add an Ansible web controller unless more than one operator, retained
  run history, or delegated credentials justify another service.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt