# ADR 063: PostHog Alerting and Lightweight Incident Management - HTML version: https://robbiepalmer.me/projects/recipe-site/adrs/063-posthog-alerting-and-slack - Project: Recipe Site (https://robbiepalmer.me/projects/recipe-site.md) - Status: Proposed - Date: 2026-07-29 ## Summary Propose using PostHog log alerts as the recipe system's operational detection, alert-state, and investigation layer, with delivery through PostHog's native Slack integration to a private channel in a dedicated workspace. Once configured and verified, PostHog and Slack will form an intentionally lightweight incident workflow for a solo founder: * application logs and traces provide the signals; * Terraform-managed PostHog alerts detect abnormal error conditions; * PostHog maintains firing, resolved, evaluation-error, and auto-disabled states; * Slack provides an isolated, high-visibility notification inbox; * each notification links back to PostHog for investigation. This ADR remains proposed until the workspace, integration, notification settings, and end-to-end firing and resolution paths have been configured and tested. It is not a claim that PostHog and Slack form a complete incident-management platform. The proposed system has no on-call schedule, acknowledgement deadline, escalation, SMS or voice paging, status page, or independently hosted uptime check. Those capabilities are unnecessary at the current team size and incident volume, but this decision defines when a dedicated error-tracking, observability, uptime, or incident-management product becomes justified. ## Context [ADR 047](/projects/recipe-site/adrs/047-posthog-logs) and [ADR 062](/projects/recipe-site/adrs/062-direct-otlp-export-to-posthog) place operational logs and traces in PostHog so failures can be correlated with users, replays, and product outcomes. That telemetry is useful for retrospective diagnosis, but without active detection it still requires the project owner to notice a problem and open PostHog manually. The recipe system has three production log alerts: | Alert | Signal | Evaluation | Purpose | | --------------------------- | -------------------------------------------- | -------------- | ----------------------------------------------------------------------- | | Recipe API sustained errors | More than 5 error or fatal logs in 5 minutes | 2 of 3 periods | Detect sustained API degradation without paging on one isolated failure | | Recipe ingestion errors | More than 0 error or fatal logs in 5 minutes | 1 of 1 period | Detect any failed background ingestion attempt | | Recipe Pages errors | More than 0 error or fatal logs in 5 minutes | 1 of 1 period | Detect failures in the comparatively quiet Pages runtime | Each alert has a 30-minute cooldown. The thresholds were simulated against the preceding seven days of production data before adoption: the API and ingestion alerts each produced one representative fire, while the Pages alert produced none. Application failures recorded on traces also emit correlated error logs, so these alerts cover the operational failures visible in both Logs and Traces without maintaining duplicate trace-alert definitions. Terraform owns the alert filters, thresholds, periods, cooldowns, and enabled state. PostHog's official Terraform provider does not yet expose the newer Logs Alert API, so the definitions use a generic REST provider with validation and deletion protection. PostHog owns evaluation scheduling, state transitions, history, and investigation links. This provides several parts of an incident-management loop, but not all of them: | Capability | Owner | Coverage | | ------------------------------------------------ | --------------------------------------------------------- | ---------------------------- | | Signal capture | PostHog Logs and Traces, with Cloudflare logs as fallback | Implemented | | Detection and thresholds | Terraform-managed PostHog log alerts | Implemented | | Alert state and history | PostHog | Implemented | | Notification transport | Native PostHog Slack integration | Proposed; not yet verified | | Notification inbox | Private Slack channel in a dedicated workspace | Proposed; not yet configured | | Investigation | PostHog filtered logs, traces, users, and session replays | Implemented | | Cloudflare platform incidents | Terraform-managed Cloudflare notifications → Slack | Proposed complementary path | | Independent availability monitoring | None | Deferred | | Acknowledgement and ownership | One owner reading Slack | Proposed informal process | | Escalation, schedules, SMS, and voice | None | Deferred | | Incident coordination, timeline, and status page | None | Deferred | PostHog log alerts support Slack, Microsoft Teams, Discord, and generic webhook destinations. Their [alert destination implementation](https://github.com/PostHog/posthog/blob/8335cbd86f934b36e96109268e1ca372fa2cb7f7/products/alerts/backend/destination_configs.py) models Slack as an installed workspace integration selected by workspace and channel identifiers. Teams, Discord, and generic webhook destinations instead require a webhook URL. That distinction matters. A generic PostHog webhook currently has fixed headers and no sender-verifiable signature or configurable authentication header. A Cloudflare Worker or hosted automation relay would therefore need to accept a high-entropy secret URL as a bearer credential. Strict schema validation, rate limiting, and rotation would reduce the impact of abuse but could not prove that a request came from PostHog. A hosted relay would move the same upstream authentication limitation to another provider rather than remove it. Email is not a preferred operational notification surface for the project owner. A dedicated Slack workspace is easier to isolate from personal communication, keep visible on desktop and mobile, and configure for high-priority channel notifications. Slack is therefore a useful operational tool in its own right rather than an otherwise unused dependency. ### Administrative Cost and Compounding Value Slack is not administration-free. The initial setup requires a workspace, a domain-controlled account and recovery path, multi-factor authentication, a private channel, review of the PostHog app's permissions, desktop and mobile clients, and carefully configured notifications. The free plan also has limited searchable history and retention. These are real ongoing account-management responsibilities. Rolling an email relay would replace that visible administration with less visible engineering and security administration: * provision and deploy another Cloudflare Worker, route or hostname, email binding, secrets, and monitoring; * design rotation and incident response for a bearer credential embedded in the webhook URL; * test authentication rejection, payload limits and schemas, every alert state, email-binding failures, secret rotation, deployment, and recovery; * monitor both the receiver and email-delivery path without recursively depending on that path for its own alerts; * maintain a dedicated high-visibility mailbox, forwarding destination, or carefully isolated inbox rules so operational email is actually noticed. Cloudflare Email Routing provides an address and forwarding, not a dedicated operational inbox. Consequently, the relay would not remove the account and attention-management problem; it would add custom infrastructure before delivering into an inbox that still needs to be made reliable and visible. The relay would also have little compounding value. It would be a single-purpose adapter around a current PostHog destination limitation: a useful temporary patch if email were mandatory, but each new responder, destination, or routing rule would increase its configuration, tests, and security surface. Slack's administration is front-loaded, while its value can compound through additional responders, private operational channels, native integrations, and incident collaboration without expanding project-owned delivery code. That makes Slack the more durable choice even though creating the workspace is initially more conspicuous than deploying a small Worker. ## Decision ### Detection and Alert State Keep the three production alert definitions in Terraform and treat PostHog as the source of truth for operational alert state. * Alert on error and fatal logs for `recipe-api`, `recipe-ingest`, and `recipe-pages`. * Preserve the different sensitivity of interactive API, background ingestion, and quiet Pages workloads rather than applying one global threshold. * Simulate material threshold changes against at least seven days of production history. * Aim for actionable, low-noise alerts; a recurring non-incident notification is a threshold or signal-quality problem, not something to tolerate indefinitely. * Link notifications to the filtered PostHog logs and alert detail so Slack remains a prompt to investigate, not a second diagnostic interface. * Keep alert definitions in code. Do not make threshold changes only in the PostHog UI because Terraform will treat them as drift. ### Slack Delivery Create a Slack workspace dedicated to project operations and connect it using PostHog's native Slack integration. This is the proposed delivery design until the adoption criteria below pass. * Register the workspace with a domain-controlled project email address rather than a personal address. * Protect the Slack and PostHog accounts with unique credentials and multi-factor authentication, preferring passkeys where available. * Create one private `#production-alerts` channel and invite only the project owner and the PostHog integration. * Review the integration's requested Slack scopes during installation and whenever they change. * Attach all managed production log alerts to that channel in PostHog. * Configure desktop and mobile Slack clients to notify for every message in the alert channel while leaving unrelated Slack notifications disabled. * Send concise alert state, affected service, threshold result, and investigation links. Keep raw logs, credentials, request headers, and unnecessary personal data in PostHog rather than copying them into Slack. * Treat Slack as notification transport and a transient inbox, not as the authoritative alert history or diagnostic data store. Keep destination attachment manual while PostHog models alert destinations as internal Hog functions without a stable Terraform resource. Record that manual step in operational setup and verify it after alert recreation or workspace integration changes. ### Cloudflare Notifications Use [Cloudflare Notifications](https://developers.cloudflare.com/notifications/) as a complementary source for failures that originate in Cloudflare or can prevent application telemetry from reaching PostHog. Route them to the same private Slack channel, or to a separate private `#cloudflare-alerts` channel if their volume or ownership later diverges. Manage the destination and policies with Cloudflare's [Terraform alerting resources](https://developers.cloudflare.com/api/terraform/resources/alerting/) where the account entitlement and provider schema allow it: * use `cloudflare_notification_policy_webhooks` for the Slack-formatted webhook destination; * use `cloudflare_notification_policy` for individual alert types and filters; * grant the routine Cloudflare Terraform token the least-privilege Notifications read and write permissions required by those resources; * initially cover production Pages deployment failures, material Cloudflare incidents affecting the services in use, certificate or service-token expiry, and relevant billing or usage thresholds; * add Workers Observability notifications only when they close a gap not already covered by the PostHog log alerts. The Slack incoming-webhook URL is itself a bearer credential. Pass it to Terraform as a sensitive value from Doppler, do not output it, restrict access to Terraform Cloud state, and rotate it if it is disclosed. This is still a smaller security and maintenance surface than a project-owned relay: the credential authorizes posting to one Slack channel rather than access to application infrastructure, and there is no receiver code, public project endpoint, schema parser, or email-delivery path to operate. Cloudflare's [webhook documentation](https://developers.cloudflare.com/notifications/get-started/configure-webhooks/) states that webhook delivery normally requires at least one Pro-or-higher zone. At the time of this proposal, the account has only a Free zone, but Cloudflare's live delivery eligibility API reports webhooks as eligible and ready. Treat that as an entitlement to verify, not a guarantee: create and test the destination before accepting this ADR. If Cloudflare rejects or later withdraws webhook delivery, keep its native email or dashboard notifications as the temporary fallback rather than building a relay solely for this path, and reconsider a plan upgrade when the alerts justify it. The repository currently uses Cloudflare Terraform provider v4.52.8. It supports notification destinations and policies, but its alert-type validation predates the current `workers_observability_alert` identifier returned by Cloudflare. Manage the policies supported by the existing provider first. Evaluate a provider v5 upgrade or a narrowly scoped API resource separately before adding a Workers Observability policy; do not broaden this documentation PR into a provider migration. ### Incident Workflow When a Slack alert arrives, the proposed response process is: 1. open the PostHog alert or filtered-log link; 2. identify the affected service, route, trace, and user impact; 3. mitigate or deploy a fix through the normal repository workflow; 4. confirm that PostHog transitions the alert to resolved; 5. adjust the alert only when the event was noisy or failed to represent a real incident. Reading the channel notification is the informal acknowledgement mechanism while there is one owner. Emoji reactions or threads may be used as personal notes, but they are not a durable incident record and the system does not claim delivery, acknowledgement, or resolution service-level objectives. ## Deliberate Implementation Boundary PostHog owns alert evaluation and state; Slack supplies notification delivery. This ADR keeps the initial notification path small; it does not prejudge whether later triage and coordination capabilities should be built or bought. The initial implementation must not include: * a custom webhook receiver, email relay, or hosted automation relay between PostHog or Cloudflare and Slack; * durable delivery storage or a retry ledger; * acknowledgement, escalation, rota, or paging policies; * polling PostHog for alert state; * copying complete log or trace payloads into Slack; * routing application or customer notifications through the operational channel. PostHog already owns alert state, cooldowns, evaluation history, and event generation. Slack owns workspace authentication and channel delivery. Introducing a relay between them would increase the number of credentials and failure modes without improving the current solo-founder workflow. A Slack bot, interactive commands, an LLM-backed triage agent, or a notification-preference UI are not prohibited by this decision. They are outside its initial scope and require a separate ADR. That ADR must compare building with buying against the agentic tooling available at the time rather than assuming a durable vendor abstraction exists. It must also define Slack permissions, PostHog data access, prompt-injection and data-exfiltration boundaries, human approval for consequential actions, cost controls, auditability, failure behaviour, and how the bot remains optional to the basic alert delivery path. ## Failure Independence Slack is hosted outside Cloudflare, so Cloudflare application-runtime failures do not remove the Slack inbox. This isolates notification transport only: recipe-code, database, or telemetry-export failures can prevent PostHog from generating an alert. It is still better transport isolation than delivering through another Cloudflare Worker. PostHog remains a shared dependency for telemetry ingestion, alert evaluation, and notification dispatch. A PostHog outage or an export failure can therefore prevent application alerts. Maintain two complementary signal paths: * **Application and workflow failures:** PostHog alert → private Slack channel. * **Cloudflare-originated incidents:** Terraform-managed Cloudflare notifications → Slack, subject to webhook entitlement verification. Cloudflare Notifications cannot evaluate arbitrary PostHog telemetry, and PostHog cannot detect a failure that prevents telemetry from reaching it. If the site later needs independently verified uptime, add one simple external monitor hosted outside Cloudflare and PostHog. Do not introduce one solely to remove a theoretical dependency before availability has meaningful business impact. ## Alternatives and More Advanced Platforms The alternatives are not interchangeable. Each category solves a different part of the operational loop that the current PostHog-centred system only partially covers. | Option | What it would improve | Costs and gaps at the current scale | Decision | | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | **PostHog alerts with native Slack delivery** | Keeps detection beside logs, traces, users, replays, and product impact; avoids a public receiver; visible and isolated mobile/desktop alert inbox | Requires workspace, identity recovery, MFA, clients, permission review, and notification administration; free-plan retention is limited; Slack is not durable incident management | **Propose; accept after setup and end-to-end verification** | | **Dedicated Cloudflare email relay** | Fixed email recipient; infrastructure could remain in the existing Cloudflare account | Requires a secret URL, Worker, route, email binding, rotation, schema and failure-path tests, deployment and security monitoring, plus a high-visibility mailbox; a single-purpose patch that compounds poorly as people and use cases grow | Reject while native Slack meets the need | | **Microsoft Teams** | Similar chat-based notification surface; strong fit if the project later adopts Microsoft 365 and Entra | PostHog's Teams destination requires a webhook URL; adds Microsoft account and workflow complexity; weaker Linux experience; no current Microsoft ecosystem benefit | Reject for the current stack | | **Discord** | Free private channels, capable desktop and mobile clients, roles, webhooks, and a strong bot ecosystem | PostHog uses a bearer webhook URL rather than an installed workspace integration; less aligned with future operational SaaS integrations and professional responder onboarding than Slack | Viable fallback, but Slack better fits the expected direction | | **Hosted automation relay** | Low-code payload transformation and access to more delivery destinations | The PostHog-facing endpoint still relies on a secret URL unless the vendor provides a native authenticated PostHog integration; adds another vendor and failure boundary | Reject | | **Cloudflare observability and notifications only** | Lowest integration effort; strong platform-native request and infrastructure context; Terraform-managed Slack delivery where eligible | Application signals remain separated from PostHog users and product impact; cannot evaluate arbitrary PostHog telemetry; same-provider blind spot; webhook entitlement and bearer Slack URL still require management | Use only as a complementary platform signal | | **Dedicated error tracking such as Sentry or Bugsnag** | Exception grouping, stack-trace triage, release health, regressions, ownership, and issue workflows | Another SDK or export path, data store, alert configuration, billing surface, and investigation UI; weaker native connection to PostHog product context | Reconsider when exception triage outgrows log alerts | | **Dedicated observability such as Grafana Cloud, Honeycomb, Datadog, or Better Stack** | Mature metrics, logs, traces, dashboards, SLOs, query languages, alert correlation, retention, and integrations | New vendor and credentials; duplicated telemetry and cost; product and user context may split across systems; more configuration to keep aligned | Reconsider when service reliability requires SLOs or deeper multi-signal analysis | | **Independent uptime monitoring such as Checkly, Better Uptime, or UptimeRobot** | External availability checks outside Cloudflare and PostHog; detects total site or telemetry-export failure | Detects symptoms rather than application causes; another service and notification policy; does not replace PostHog investigation | Add when availability has meaningful user or revenue impact | | **On-call and incident management such as PagerDuty, incident.io, Rootly, or Opsgenie** | Durable routing, acknowledgement, escalation, schedules, SMS or voice paging, incident timelines, collaboration, audit history, and status flows | Designed for teams and rotations; cost and setup are disproportionate for one owner; still needs PostHog or another monitoring source | Reconsider with a second responder or formal response objectives | | **Build a custom incident platform** | Exact fit and full control | Recreates alert history, retries, acknowledgement, escalation, schedules, and UI; largest reliability and maintenance burden | Reject | Moving to a dedicated platform would be beneficial when it replaces real manual work or closes a measured reliability gap. Adopting one merely because it is more specialised would split operational context and add configuration without improving current incident outcomes. WhatsApp and Twilio are not retained as separate alternatives. Slack already supplies the desired high-visibility mobile notifications without their additional business-messaging onboarding, credentials, templates, policy constraints, and delivery costs, so they offer no relevant advantage for this decision. ## Consequences ### Positive * Once adopted, the system actively surfaces detected production failures instead of relying on dashboard inspection. * Alert thresholds and filters are reviewable, reproducible, and protected from accidental deletion. * The project owner receives actionable alerts in an isolated inbox configured for high visibility. * PostHog remains the single alert-evaluation and investigation surface. * The native integration avoids maintaining a public receiver, secret webhook URL, relay code, and email-delivery configuration, tests, and security monitoring. * Slack is independent of Cloudflare application deployments and runtime failures. * The workspace can later onboard other responders and useful integrations without expanding project-owned notification infrastructure. * PostHog and Cloudflare alerts can share one high-visibility operational inbox while retaining independent detection paths. * The design can later route alerts to a dedicated incident platform without changing detection thresholds. ### Negative * Log-count thresholds cannot perform every form of anomaly detection, uptime check, metric alert, or exception grouping supplied by mature dedicated products. * The workflow has no durable acknowledgement, escalation, or formal incident record. * Slack adds another account, client, vendor dependency, and permission grant. * The workspace needs account recovery, multi-factor authentication, membership, channel, integration-permission, and notification administration. * Slack's free-plan searchable history and retention are limited. * PostHog destination attachment remains a manual configuration step. * Cloudflare-to-Slack delivery uses a bearer webhook URL that Terraform Cloud state and Cloudflare must store, protect, and support rotating. * Cloudflare webhook entitlement on the current Free account must be verified despite the live API reporting it as eligible, and the older provider cannot express every current alert type. * A PostHog outage or telemetry-export failure can impair both detection and delivery. * Slack notification delivery is not guaranteed paging and can be affected by client settings, connectivity, or account access. ## Adoption Criteria Keep this ADR in `Proposed` status until all of the following have been completed: * create the dedicated Slack workspace using a domain-controlled project identity with a tested account-recovery path; * enable multi-factor authentication or a passkey on the Slack and PostHog accounts; * create the private `#production-alerts` channel, restrict membership, install PostHog's native Slack integration, and review its granted scopes; * attach all three managed production alerts to the channel and record the manual reattachment procedure; * configure and verify high-visibility desktop and mobile notifications for the channel, including behaviour during the project's chosen do-not-disturb schedule; * produce production-safe synthetic signals for the API, ingestion, and Pages alerts and verify that their firing and resolved notifications reach Slack; * verify that notification links open the correct PostHog alert and filtered investigation views; * confirm that no raw secrets, request headers, full payloads, or unnecessary personal data are copied into Slack; * grant the Terraform Cloudflare token least-privilege Notifications permissions and create the Slack webhook destination and selected policies through Terraform; * verify Terraform Cloud treats the Slack webhook URL as sensitive, restrict state access, and document its rotation procedure; * trigger at least one Cloudflare test notification and verify that it reaches Slack; if webhook entitlement fails on the current plan, record the fallback and leave the ADR proposed; * verify the initial Pages, incident, certificate or token, and billing or usage policies against the account's live available-alert list rather than assuming every provider enum is entitled; Record the verification outcome and change the ADR status to `Accepted` in a follow-up commit only after every criterion passes. Until then, the existing PostHog alert definitions remain implemented, but Slack delivery and the incident workflow described here are not operational. ## Guardrails and Verification * Simulate new and materially changed alerts against recent production history before enabling them. * Review firing frequency and investigate repeated false positives rather than muting them indefinitely. * Periodically verify that each managed alert is enabled and attached to the private Slack channel. * Protect the Slack account with multi-factor authentication and review active sessions and integration scopes periodically. * Keep the alert channel private and restrict workspace membership. * Include only concise operational metadata and PostHog links in Slack messages. * Keep secrets, authentication data, complete request payloads, and unnecessary personal data out of Slack. * Configure and test desktop and mobile notifications specifically for the alert channel. * Trigger a synthetic alert after initial setup and after material integration changes; verify both firing and resolved messages reach Slack and their PostHog links work. * Keep Cloudflare notification policies in Terraform, test their Slack destination independently, and compare configured alert types periodically with Cloudflare's live available-alert list. * Keep any future bot or LLM triage path optional; a bot failure must not prevent PostHog or Cloudflare from posting the original notification. ## When To Revisit Reconsider the delivery design when any of these becomes true: * Slack alerts are routinely missed, delayed, or noisy; * PostHog alert evaluation, history, or investigation is insufficient for recurring incidents; * exception grouping and regression detection require dedicated error tracking; * service-level objectives require metrics, burn-rate alerts, or independent availability checks; * another person joins the operational rota; * acknowledgement, escalation, schedules, SMS, voice calls, or incident timelines become necessary; * the number of services, destinations, or notification policies makes fixed routing unmanageable; * a PostHog or Cloudflare failure creates a commercially material monitoring blind spot; * the project adopts Microsoft 365, another messenger, or an incident-management service as a core platform; * PostHog exposes stable destination resources that Terraform can safely reconcile. --- Markdown index of this site: https://robbiepalmer.me/llms.txt