# ADR 010: Layered observability for shared agent workspaces

- HTML version: https://robbiepalmer.me/projects/agent-friendly-remote-development/adrs/010-layered-workspace-observability
- Project: Agent-friendly Remote Development (https://robbiepalmer.me/projects/agent-friendly-remote-development.md)
- Status: Proposed
- Date: 2026-09-12
- Initiatives: Semi-autonomous Software Development (https://robbiepalmer.me/initiatives/semi-autonomous-software-development.md)

# Context

This project needs telemetry at three levels. It must show the remote machine's
health, the state and resource use of each K3s workspace, and the lifecycle of
coding-agent sessions launched through T3 Code. An alert should name the
affected workspace and distinguish host pressure, a Kubernetes action, and an
application or child-process failure.

The first deployment has one small host and two users. A self-hosted
Prometheus, Grafana, Loki, and Tempo stack would consume memory and create more
stateful services than this deployment warrants. The chosen system should
still use standard Kubernetes and OpenTelemetry signals so the project can
move to that stack later without replacing its instrumentation.

Kubernetes documents
[kube-state-metrics](https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/)
as the service that converts Kubernetes API object state into Prometheus
metrics. Its pod metrics include namespace-labelled restart counts and the
last termination reason and exit code. The last-termination series remain
[experimental](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md),
so the project must pin the exporter and test every series used by an alert.

T3 Code writes rotating local trace files and can export traces and metrics
over OTLP. Its current instrumentation covers provider sessions and turns,
orchestration commands, terminal sessions, git commands, and outcome labels,
as described in its
[observability documentation](https://github.com/pingdotgg/t3code/blob/main/docs/operations/observability.md).

# Decision

Create a layered observability system. Use Netdata for infrastructure and
application metrics, PostHog for T3 Code logs and traces, Healthchecks.io for
external host-loss detection, and a dedicated private Slack channel as the
alert inbox.

## Netdata dashboard and alerts

Install one Netdata Agent as a NixOS service on the remote host. It will store
its own history and serve its own dashboard through the project's private
Tailscale network. Do not expose the dashboard publicly, configure parent
streaming, or attach it to Netdata Cloud.

The Agent will collect host, systemd, filesystem, pressure-stall, and K3s
cgroup metrics. Enable its eBPF OOM collector and add a local alert for OOM
events because Netdata does not ship one by default. This signal covers a
kernel OOM kill of an agent child process that leaves the T3 Code container
running. Add a small local collector for the existing ext4 project quotas so
byte and inode use carry the same workspace ID. Netdata's
[Kubernetes cgroup collector](https://learn.netdata.cloud/docs/collecting-metrics/collectors/containers-and-vms/kubernetes-containers)
provides namespace, pod, container, memory-limit, CPU-throttling, I/O, PID, and
pressure labels. Its
[eBPF OOM collector](https://learn.netdata.cloud/docs/collecting-metrics/operating-systems/ebpf-oomkill)
records OOM kills by cgroup and systemd service.

Configure Agent-dispatched Slack notifications with a project-specific
incoming webhook stored in this project's secret store. Netdata documents
[direct Slack delivery from the Agent](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications/slack),
so this path needs neither Netdata Cloud nor parent streaming.

## Kubernetes state

Deploy one pinned kube-state-metrics instance in a dedicated `observability`
namespace with read-only RBAC. Netdata will scrape its Prometheus endpoint.
This adds Kubernetes lifecycle facts that cgroups cannot provide, including
`OOMKilled`, `Evicted`, `CrashLoopBackOff`, readiness, and restart counts. Keep
the exporter outside both user namespaces and give it a small resource request
and limit.

## T3 Code telemetry

Run one pinned OpenTelemetry Collector Contrib service on the host. T3 Code in
both workspaces will send OTLP/HTTP to its cluster-private endpoint. The
Collector will:

* send T3 Code metrics to Netdata's loopback OTLP/gRPC receiver;
* send T3 Code traces and selected T3 Code container logs to this project's
  PostHog project;
* add a stable workspace ID from the service name or Kubernetes namespace;
* remove disallowed attributes before data leaves the host; and
* expose its own health and dropped-record metrics to Netdata.

Netdata accepts
[OTLP/gRPC rather than OTLP/HTTP](https://learn.netdata.cloud/docs/opentelemetry/otlp-ingestion),
so the Collector bridges the protocol used by T3 Code. Its host placement also
keeps it available during a K3s restart. Bind the incoming OTLP port only to
the K3s bridge or firewall it to pod traffic.

Pin or upgrade the T3 Code image to a release that contains the documented
OTLP exporters. The deployed image, rather than upstream `main`, must pass the
telemetry smoke test.

Create a PostHog project for this remote development environment. Give it a
project token, dashboards, retention and billing controls, alert definitions,
and Slack destination.

## Host-loss detection

A Netdata Agent running on the monitored host cannot deliver an alert after
the host or its network path has failed. Add one Healthchecks.io check as an
external dead man's switch. A NixOS systemd timer will send a short heartbeat
with a bounded timeout. Healthchecks.io will notify the project's Slack
channel when the heartbeat becomes late and again when it recovers. Its
[systemd guidance](https://healthchecks.io/docs/monitoring_systemd_tasks/)
explicitly covers whole-machine failure, and its free Hobbyist plan currently
allows [20 checks](https://healthchecks.io/pricing/).

Treat the Healthchecks.io ping URL as a secret. The check is the only off-host
dependency used to determine whether the remote machine itself is reachable.

## Attribution and data handling

Use stable, non-personal workspace identifiers such as `operator` and `pilot`.
Apply them consistently as the Kubernetes namespace label,
`T3CODE_OTLP_SERVICE_NAME`, `service.namespace`, and `workspace.id`. PostHog
may retain a session or trace ID for investigation. Slack messages must contain
only the workspace ID, failure class, affected component, time, and a link to
the relevant project-owned view.

Use the following ownership split:

| Question                                                                     | Source of truth                                 | Required attribution                             |
| ---------------------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------ |
| Is the host, K3s service, disk, or network under pressure?                   | Project Netdata Agent                           | host and affected service                        |
| Which workspace is near a CPU, memory, PID, byte, or inode limit?            | Netdata cgroups plus a quota collector          | workspace ID and limit                           |
| Why did a workspace container restart?                                       | kube-state-metrics                              | namespace, pod, container, reason, and exit code |
| Did the kernel kill an agent child process without restarting the container? | Netdata eBPF OOM data plus the kernel journal   | cgroup, process, and mapped workspace ID         |
| Which provider session or turn failed?                                       | T3 Code OTLP metrics, traces, and selected logs | workspace ID, provider, outcome, and trace ID    |
| Has the entire remote host disappeared?                                      | Healthchecks.io                                 | host and last heartbeat time                     |

The Collector must use an allowlist before exporting to PostHog. Do not export
prompts, responses, file contents, diffs, terminal output, environment
variables, credentials, provider event payloads, or full working-directory
paths. Apply a default-deny filter to stdout and drop an unrecognised log line
rather than risk exporting its body. Keep T3 Code's rotating local trace file
and K3s container log rotation as fallbacks. The pilot user must know what the
system records before their workspace is enabled.

Alert immediately on an OOM kill, `OOMKilled`, eviction, crash loop, failed K3s
service, exhausted quota, failed telemetry pipeline, or missed external
heartbeat. Alert on sustained memory pressure, CPU throttling, disk pressure,
and repeated provider failures only after a short window. Every alert must
send a recovery message.

# Acceptance criteria

Keep this ADR Proposed until a disposable test proves all of the following:

1. The Netdata dashboard works through its private URL without parent streaming
   or Netdata Cloud.
2. Netdata shows separate host, `operator`, and `pilot` CPU, memory, pressure,
   throttling, PID, filesystem, byte-quota, and inode-quota views.
3. The deployed T3 Code image exports successful and failed provider turns, a
   terminal lifecycle event, and the documented counters through the Collector
   without losing its local trace file.
4. Controlled child-process and container OOM tests name the correct workspace
   and cause. The other workspace remains usable.
5. A non-OOM provider child-process failure appears in PostHog with its
   workspace ID and trace. Repeated failures trigger Slack without requiring a
   pod restart.
6. Stopping K3s and stopping kube-state-metrics produce distinct alerts.
7. Making the host unreachable causes Healthchecks.io to alert Slack. Recovery
   produces one matching notification.
8. Stopping the Collector or either export path produces a pipeline alert
   rather than a silent gap.
9. A review of exported records finds none of the disallowed prompt, source,
   terminal, path, or credential data.
10. Netdata, kube-state-metrics, and the Collector stay below 512 MiB resident
    memory and 5 percent of one CPU at steady state during concurrent agent
    work. Each component also has a bounded memory policy.
11. Seven days of pilot use produce no duplicate or unactionable pages. The
    sample records PostHog volume and projected cost, with a billing limit that
    prevents unplanned spend.

# Alternatives

A separate Netdata Cloud space would provide off-host access and node-loss
detection. It would reintroduce a hosted Netdata control plane and its plan
limits when the Agent already serves a private dashboard. The small external
heartbeat covers the one failure the local Agent cannot report.

`kube-prometheus-stack` would provide the familiar Kubernetes package of
Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics. It
duplicates Netdata's metric storage, dashboard, and alerting and adds more
stateful services to the host. Adopt it when the project grows beyond one
cluster, needs shared PromQL recording rules or SLOs, or Netdata can no longer
scrape the required metrics.

Netdata's native Kubernetes state collector could replace kube-state-metrics
and remove one small pod. This decision keeps the exporter because its metric
names and labels are Kubernetes community conventions and can move to
Prometheus or another compatible backend later. The cost is a pinned component
and extra alert tests for its experimental last-termination series.

Netdata alone would cover host pressure, cgroups, and container termination.
It would miss the trace of a failed T3 Code provider turn and could not report
complete host loss. PostHog alone has the inverse problem. An application
cannot report the kernel killing it, and a host outage can stop export before
it records the cause.

# Consequences

An incident can be checked at each level. Netdata shows machine and workspace
pressure, kube-state-metrics records what K3s did, and PostHog shows what T3
Code and the agent were doing. Healthchecks.io covers complete host loss.

The project adds one small Kubernetes exporter, two host services, and one
hosted heartbeat check. It must patch and monitor those components. The
Collector needs redaction rules, while eBPF collection needs narrowly scoped
host privileges.

Local Netdata history becomes unavailable while the remote host is down and
may disappear with an unrecoverable host disk. The external heartbeat proves
that the host vanished, not why. Add an off-host metrics store or a dedicated
Netdata parent inside this project's boundary if that gap prevents incident
diagnosis often enough to justify another service.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt