# ADR 010: Layered observability for agent workspaces

- HTML version: https://robbiepalmer.me/projects/agent-friendly-remote-development/adrs/010-layered-workspace-observability
- Project: Agent-friendly Remote Development (https://robbiepalmer.me/projects/agent-friendly-remote-development.md)
- Status: Proposed
- Date: 2026-09-12
- Initiatives: Semi-autonomous Software Development (https://robbiepalmer.me/initiatives/semi-autonomous-software-development.md)

# Context

This project needs telemetry at three levels. It must show the remote machine's
health, the state and resource use of each K3s workspace, and the lifecycle of
coding-agent sessions launched through T3 Code. An alert should name the
affected workspace and distinguish host pressure, a Kubernetes action, and an
application or child-process failure.

The first deployment has one small host and one user. A future multi-user
cluster will use dedicated workspace nodes. A self-hosted
Prometheus, Grafana, Loki, and Tempo stack would consume memory and create more
stateful services than this deployment warrants. The chosen system should
still use standard Kubernetes and OpenTelemetry signals so the project can
move to that stack later without replacing its instrumentation.

Kubernetes documents
[kube-state-metrics](https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/)
as the service that converts Kubernetes API object state into Prometheus
metrics. Its pod metrics include namespace-labelled restart counts and the
last termination reason and exit code. The last-termination series remain
[experimental](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md),
so the project must pin the exporter and test every series used by an alert.

T3 Code writes rotating local trace files and can export traces and metrics
over OTLP. Its current instrumentation covers provider sessions and turns,
orchestration commands, terminal sessions, git commands, and outcome labels,
as described in its
[observability documentation](https://github.com/pingdotgg/t3code/blob/main/docs/operations/observability.md).

# Decision

Create a layered observability system. Use Netdata for infrastructure and
application metrics, PostHog for T3 Code logs and traces, a managed heartbeat
service for external host-loss detection, and a dedicated private Slack
channel as the alert inbox.

## Netdata dashboard and alerts

Install one Netdata Agent as a NixOS service on the remote host. It will store
its own history and serve its own dashboard through the project's private
Tailscale network. Do not expose the dashboard publicly, configure parent
streaming, or attach it to Netdata Cloud.

The Agent will collect host, systemd, filesystem, pressure-stall, and K3s
cgroup metrics. Enable its eBPF OOM collector and add a local alert for OOM
events because Netdata does not ship one by default. This signal covers a
kernel OOM kill of an agent child process that leaves the T3 Code container
running. Add a small local collector for the existing ext4 project quotas so
byte and inode use carry the same workspace ID. Netdata's
[Kubernetes cgroup collector](https://learn.netdata.cloud/docs/collecting-metrics/collectors/containers-and-vms/kubernetes-containers)
provides namespace, pod, container, memory-limit, CPU-throttling, I/O, PID, and
pressure labels. Its
[eBPF OOM collector](https://learn.netdata.cloud/docs/collecting-metrics/operating-systems/ebpf-oomkill)
records OOM kills by cgroup and systemd service.

Configure Agent-dispatched Slack notifications with a project-specific
incoming webhook stored in this project's secret store. Netdata documents
[direct Slack delivery from the Agent](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications/slack),
so this path needs neither Netdata Cloud nor parent streaming.

## Kubernetes state

Deploy one pinned kube-state-metrics instance in a dedicated `observability`
namespace with read-only RBAC. Netdata will scrape its Prometheus endpoint.
This adds Kubernetes lifecycle facts that cgroups cannot provide, including
`OOMKilled`, `Evicted`, `CrashLoopBackOff`, readiness, and restart counts. Keep
the exporter outside both user namespaces and give it a small resource request
and limit.

## T3 Code telemetry

Run one pinned OpenTelemetry Collector Contrib service on the current host. T3
Code will send OTLP/HTTP to its private endpoint. A future multi-node cluster
will run an equivalent collector outside user namespaces. The
Collector will:

* send T3 Code metrics to Netdata's loopback OTLP/gRPC receiver;
* send T3 Code traces and selected T3 Code container logs to the existing
  shared PostHog EU project;
* add a stable workspace ID from the service name or Kubernetes namespace;
* remove disallowed attributes before data leaves the host; and
* expose its own health and dropped-record metrics to Netdata.

Netdata accepts
[OTLP/gRPC rather than OTLP/HTTP](https://learn.netdata.cloud/docs/opentelemetry/otlp-ingestion),
so the Collector bridges the protocol used by T3 Code. Its host placement also
keeps it available during a K3s restart. Bind the incoming OTLP port only to
the K3s bridge or firewall it to pod traffic.

Pin or upgrade the T3 Code image to a release that contains the documented
OTLP exporters. The deployed image, rather than upstream `main`, must pass the
telemetry smoke test.

Reuse the existing PostHog EU project that already contains telemetry for the
personal site and other personal applications. Mark every remote-development
record with `deployment.environment.name=remote-development` and a
`workspace.id`. Every dashboard, saved insight, alert, and Slack notification
for this system must filter on the environment marker. Views and alerts for one
workspace must also filter on its workspace ID.

Use the shared project's ingestion token and trace and log retention. Keep
trace and log retention at 30 days or less. Prefix saved assets with
`Remote development /`, route matching alerts to the project-owned Slack
destination, and keep the account-level billing controls enabled.

## Host-loss detection

A Netdata Agent running on the monitored host cannot deliver an alert after
the host or its network path has failed. Add one check at a managed heartbeat
service that runs outside the host and its cloud provider. A NixOS systemd
timer will send a short outbound HTTPS heartbeat with a bounded timeout. The
service must notify the project's Slack channel when the heartbeat becomes
late and again when it recovers.

Treat the ping URL as a secret. The heartbeat provider is selected separately
because it is an independently replaceable dependency. This ADR owns the need
for the signal and its data boundary, not the provider choice.

## Attribution and data handling

Use stable, non-personal workspace identifiers such as `operator` and an
opaque workspace ID.
Apply them consistently as the Kubernetes namespace label,
`T3CODE_OTLP_SERVICE_NAME`, `service.namespace`, and `workspace.id`. PostHog
may retain a session or trace ID for investigation. Slack messages must contain
only the workspace ID, failure class, affected component, time, and a link to
the relevant filtered view.

Use the following ownership split:

| Question                                                                     | Source of truth                                 | Required attribution                             |
| ---------------------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------ |
| Is the host, K3s service, disk, or network under pressure?                   | Project Netdata Agent                           | host and affected service                        |
| Which workspace is near a CPU, memory, PID, byte, or inode limit?            | Netdata cgroups plus a quota collector          | workspace ID and limit                           |
| Why did a workspace container restart?                                       | kube-state-metrics                              | namespace, pod, container, reason, and exit code |
| Did the kernel kill an agent child process without restarting the container? | Netdata eBPF OOM data plus the kernel journal   | cgroup, process, and mapped workspace ID         |
| Which provider session or turn failed?                                       | T3 Code OTLP metrics, traces, and selected logs | workspace ID, provider, outcome, and trace ID    |
| Has the entire remote host disappeared?                                      | Managed heartbeat service                       | host and last heartbeat time                     |

The Collector must use an allowlist before exporting to PostHog. Do not export
prompts, responses, file contents, diffs, terminal output, environment
variables, credentials, provider event payloads, or full working-directory
paths. Apply a default-deny filter to stdout and drop an unrecognised log line
rather than risk exporting its body. Keep T3 Code's rotating local trace file
and K3s container log rotation as fallbacks. Any later user must know what the
system records before their workspace is enabled.

Alert immediately on an OOM kill, `OOMKilled`, eviction, crash loop, failed K3s
service, exhausted quota, failed telemetry pipeline, or missed external
heartbeat. Alert on sustained memory pressure, CPU throttling, disk pressure,
and repeated provider failures only after a short window. Every alert must
send a recovery message.

# Acceptance criteria

Keep this ADR Proposed until a disposable test proves all of the following:

1. The Netdata dashboard works through its private URL without parent streaming
   or Netdata Cloud.
2. Netdata shows separate host and `operator` CPU, memory, pressure, throttling,
   PID, filesystem, byte-quota, and inode-quota views. The multi-node evaluation
   repeats this with a second opaque workspace ID.
3. The deployed T3 Code image exports successful and failed provider turns, a
   terminal lifecycle event, and the documented counters through the Collector
   without losing its local trace file.
4. Controlled child-process and container OOM tests name the correct workspace
   and cause. The other workspace remains usable.
5. A non-OOM provider child-process failure appears in the filtered PostHog
   views with its workspace ID and trace. Repeated failures trigger Slack
   without requiring a pod restart, while unrelated application records do
   not trigger the alert.
6. Stopping K3s and stopping kube-state-metrics produce distinct alerts.
7. Making the host unreachable causes the managed heartbeat service to alert
   Slack. Recovery produces one matching notification.
8. Stopping the Collector or either export path produces a pipeline alert
   rather than a silent gap.
9. A review of exported records finds none of the disallowed prompt, source,
   terminal, path, or credential data.
10. Netdata, kube-state-metrics, and the Collector stay below 512 MiB resident
    memory and 5 percent of one CPU at steady state during concurrent agent
    work. Each component also has a bounded memory policy.
11. Seven days of representative use produce no duplicate or unactionable
    pages. The sample records PostHog volume and projected cost, with a billing
    limit that prevents unplanned spend.

# Alternatives

A dedicated PostHog project would enforce the boundary through separate
project settings and credentials. The existing PostHog project already keeps
several personal applications separate with record filters and independent
Slack destinations. The first deployment has one operator, and the Collector
removes disallowed data before export, so another PostHog project adds little
protection for this pilot. Revisit this choice before another person's
workspace is enabled or if remote-development telemetry needs different
retention, access, or billing controls.

A separate Netdata Cloud space would provide off-host access and node-loss
detection. It would reintroduce a hosted Netdata control plane and its plan
limits when the Agent already serves a private dashboard. The small external
heartbeat covers the one failure the local Agent cannot report.

`kube-prometheus-stack` would provide the familiar Kubernetes package of
Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics. It
duplicates Netdata's metric storage, dashboard, and alerting and adds more
stateful services to the host. Adopt it when the project grows beyond one
cluster, needs shared PromQL recording rules or SLOs, or Netdata can no longer
scrape the required metrics.

Netdata's native Kubernetes state collector could replace kube-state-metrics
and remove one small pod. This decision keeps the exporter because its metric
names and labels are Kubernetes community conventions and can move to
Prometheus or another compatible backend later. The cost is a pinned component
and extra alert tests for its experimental last-termination series.

Netdata alone would cover host pressure, cgroups, and container termination.
It would miss the trace of a failed T3 Code provider turn and could not report
complete host loss. PostHog alone has the inverse problem. An application
cannot report the kernel killing it, and a host outage can stop export before
it records the cause.

# Consequences

An incident can be checked at each level. Netdata shows machine and workspace
pressure, kube-state-metrics records what K3s did, and PostHog shows what T3
Code and the agent were doing. The managed heartbeat service covers complete
host loss.

The project adds one small Kubernetes exporter, two host services, and one
hosted heartbeat check. It must patch and monitor those components. The
Collector needs redaction rules, while eBPF collection needs narrowly scoped
host privileges.

PostHog configuration errors can mix remote-development records into unrelated
views or send the wrong Slack notification. The stable environment filter,
workspace filter, and asset prefix are required configuration. The shared
project also means trace and log retention and billing limits cannot vary
independently for this system.

Local Netdata history becomes unavailable while the remote host is down and
may disappear with an unrecoverable host disk. The external heartbeat proves
that the host vanished, not why. Add an off-host metrics store or a dedicated
Netdata parent inside this project's boundary if that gap prevents incident
diagnosis often enough to justify another service.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt