# ADR 056: Stateful Workers AI Code Review

- HTML version: https://robbiepalmer.me/projects/personal-site/adrs/056-stateful-workers-ai-code-review
- Project: Personal Site (https://robbiepalmer.me/projects/personal-site.md)
- Status: Proposed
- Date: 2026-07-25
- Supersedes: personal-site:046-custom-agentic-code-review

# Context

[ADR 046](/projects/personal-site/adrs/046-custom-agentic-code-review)
established that a focused, custom reviewer is economically credible. In
practice it produced better findings per dollar than Qodo, CodeRabbit,
Greptile, or the other evaluated services. Its weakness is now orchestration:
the GitHub Actions implementation is stateless and fires on every push,
re-sending largely unchanged diffs and repository context. Commit frequency
therefore drives token spend even when a new commit does not justify another
full review.

The managed alternatives have also weakened. Gemini's consumer GitHub reviewer
has shut down and now continues only as an enterprise product
([ADR 041](/projects/personal-site/adrs/041-gemini-code-assist)). Qodo's free
integration produced no useful activity, and its useful trial ended
([ADR 045](/projects/personal-site/adrs/045-qodo)). Paying for Qodo remains
possible, but observed value per dollar was below the custom reviewer.

Meanwhile, inference is becoming more competitive: strong closed and
open-weight coding models are being released rapidly across providers.
Review-specific orchestration, repository policy, retained state, and
evaluation data are therefore more durable investments than coupling the
workflow to today's best model or review SaaS.

# Decision

I propose moving the custom reviewer from per-push GitHub Actions to a
**stateful service on Cloudflare Workers**:

1. A Worker receives verified GitHub webhooks and applies a deterministic
   eligibility policy before invoking a model.
2. One SQLite-backed
   [Durable Object](https://developers.cloudflare.com/durable-objects/) per Pull
   Request serialises events and stores the reviewed head SHA, changed-hunk
   fingerprints, prior findings and dispositions, prompt/model versions, and
   token and cost metrics.
3. An alarm coalesces rapid pushes. A
   [Cloudflare Workflow](https://developers.cloudflare.com/workflows/) performs
   the retryable fetch, scout, merge, and GitHub publication steps; all steps
   are idempotent against the PR state.
4. Workers AI is the default inference path. Model identifiers remain
   configuration selected by the existing scorecard, not architecture.
   OpenRouter remains an escape hatch when a materially better model is not
   available on Workers AI.
5. Each run appends a versioned analytical record to R2, forming a small data
   lake separate from the Durable Object's operational state. It captures task
   and change characteristics, originating coding agent where known, model and
   prompt versions, findings and dispositions, latency, tokens, cache hits, and
   cost.

# Architecture

The latency-sensitive review path and the slower evaluation path are separate:

```mermaid
flowchart TB
GH["GitHub Pull Request events"]

subgraph ONLINE["Review loop — latency sensitive"]
W["Webhook Worker"]
DO[("Pull Request Durable Object")]
WF["Review Workflow"]
AI["Workers AI / OpenRouter"]
R2[("R2 review data lake")]

W --> DO
DO -->|"debounce + risk policy"| WF
WF --> AI
AI --> WF
WF --> R2
end

subgraph OFFLINE["Evaluation and learning — asynchronous"]
SIG["Later commits, thread events, replies, merge state"]
EV["Evaluator Workflow + evaluation agent"]
OUT["Attributed outcomes + scorecard"]
CFG["Model, prompt, and trigger policy"]
LEARN["Tests, AGENTS.md, skills, coding-agent choice"]

SIG --> EV
EV --> OUT
OUT --> CFG
OUT --> LEARN
end

GH --> W
WF -->|"publish or update findings"| GH
GH --> SIG
R2 --> EV
CFG -. "versioned improvements" .-> WF
```

The reviewer will run at the **Pull Request level, not the commit level**:

* run a full review when a non-draft PR becomes ready, on an explicit command
  or label, and before merge when material changes have not been reviewed;
* run an incremental review after a quiet period when changed hunks are
  materially different, touch security- or data-sensitive paths, or invalidate
  an unresolved finding;
* skip superseded webhook deliveries, already-reviewed content, generated
  files, lockfile-only changes, and small non-semantic edits unless explicitly
  requested; and
* update existing findings instead of publishing duplicate threads.

Incremental prompts will contain the stable review policy first, followed by
bounded relevant context, unresolved findings, and only the new or modified
hunks. The same PR/model pair will use a stable session-affinity key.
[Workers AI prefix caching](https://developers.cloudflare.com/workers-ai/features/prompt-caching/)
can then bill supported cached prefixes at a discount, while the durable state
prevents unnecessary calls and unnecessary context from being sent at all.
These are separate optimisations: Durable Objects do not themselves cache model
computation, and prefix-cache hits are not guaranteed.

# Evaluation and Success Metrics

An asynchronous evaluator runs when a Pull Request is merged or closed, and
again after a configurable outcome window if evidence remains incomplete. It
joins each finding to subsequent diffs, thread resolution, replies and
reactions, and final PR state. Deterministic evidence is preferred; a separate
evaluation agent adjudicates ambiguous cases and must record its evidence and
confidence. It does not need to block review. High-confidence outcomes may feed
the live PR memory sooner so an already-addressed finding is not repeated.

Each finding receives one versioned outcome:

* **confirmed-fixed** — a later commit addresses the cited issue, supported by
  the changed hunk and preferably thread resolution or acknowledgement;
* **acknowledged** — the author accepts the issue but it is deferred or resolved
  outside an attributable code change;
* **rejected** — the author explicitly explains that it is wrong, irrelevant,
  or not worth acting on;
* **superseded** — the affected code disappears or changes for another reason,
  so reviewer impact cannot be inferred; or
* **no observable response** — the finding is unresolved or silently ignored
  when the window closes. This is a workflow outcome, not evidence that the
  finding was correct or incorrect.

The scorecard is calculated at finding, review-run, Pull Request, model, prompt,
and originating-agent levels:

| Metric                      | Calculation                                                                                                              |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Adjudicated acceptance rate | `(confirmed-fixed + acknowledged) / (confirmed-fixed + acknowledged + rejected)`; excludes censored outcomes             |
| Fix-through rate            | `confirmed-fixed / published findings`, reported with the no-response share                                              |
| Noise rate                  | `(rejected + duplicate findings) / published findings`                                                                   |
| Cost per accepted finding   | Total review, evaluation, runtime, and storage cost divided by `confirmed-fixed + acknowledged`                          |
| Token efficiency            | Accepted findings per million uncached input tokens, alongside cache-hit rate                                            |
| Review efficiency           | Model calls and uncached input tokens per PR versus the stateless GitHub Actions baseline                                |
| Time to useful finding      | PR-ready timestamp to the first subsequently accepted finding                                                            |
| Missed-defect proxy         | Downstream human, CI, rollback, or incident findings in review scope that no reviewer raised; useful but not true recall |
| Upstream prevention         | Recurrence rate for a failure class before and after adding a test, `AGENTS.md` rule, skill, or agent change             |

Model and coding-agent comparisons must be stratified by change size, risk,
language, task type, and repository area; otherwise task mix will dominate the
result. Important model decisions should also use controlled replays of a
frozen PR set with the same context, prompt, and budget. The baseline window,
minimum sample size, and target improvements must be fixed before evaluating
the proposal rather than selected after seeing results.

The service remains advisory during evaluation. It becomes **Accepted** only
when replaying representative PR histories and then running live PRs shows:

* materially fewer model calls and uncached input tokens per PR than the
  GitHub Actions baseline;
* lower cost per accepted finding, with improved or non-inferior adjudicated
  acceptance and noise rates;
* no material loss of coverage from batching and incremental review; and
* maintenance effort and total spend remain below the value of buying Qodo or
  another managed reviewer.

# Strategic Considerations

* **Own the compounding layer.** Prompts, review policy, PR memory, accepted and
  rejected findings, and cost/quality measurements improve future model
  selection. A vendor subscription does not make that evidence portable.
* **Improve the development loop upstream.** The review corpus can expose
  recurring failure classes by code area, task type, or originating agent.
  Stable patterns should become deterministic tests or linters where possible,
  then `AGENTS.md` guidance, reusable skills, or specialised agents. The
  objective is not only better review; it is preventing the same defect from
  reaching review again.
* **Create a private, relevant benchmark.** Longitudinal performance on real
  changes and human dispositions is more useful to this workflow than relying
  only on saturated public coding benchmarks. The same evidence can guide both
  reviewer-model selection and which coding agent to use across Claude Code,
  Codex, OpenCode, and future tools.
* **Exploit model competition.** Provider and model adapters must keep the
  structured finding schema stable so new closed or open-weight models can be
  benchmarked and promoted without redesigning the service.
* **Spend by risk, not activity.** Deterministic triage and a cheap scout should
  handle routine deltas; the multi-scout and merger path is reserved for
  material or high-risk changes.
* **Preserve an exit.** Cloudflare consolidates webhook compute, state,
  inference, observability, and existing project infrastructure, but it becomes
  a larger platform dependency. The provider-neutral inference boundary and
  stored scorecard preserve migration to OpenRouter, direct APIs, or another
  runtime.
* **Create reusable review infrastructure.** Webhook ingestion, durable
  context, policy-based triggering, multi-model review, suggestion delivery,
  and feedback capture generalise beyond code. Future products could reuse
  these primitives for content review—for example, suggesting recipe
  improvements or adapting public recipes to a user's diet and preferences.
  Each product still requires its own ADR for domain policy, safety, privacy,
  and user experience.
* **Keep the scope narrow.** This is one repository's review loop, not a
  general code-review platform. Dashboards, repository-wide semantic indexing,
  content-review features, compliance policy, and team administration remain
  out of scope; the first implementation should only preserve clean reusable
  boundaries.

# Alternatives

## Tune the Existing GitHub Action

GitHub concurrency, path filters, labels, and scheduled runs could reduce
invocations cheaply. They do not provide authoritative cross-run PR memory,
finding lifecycle, or efficient incremental context, so the workflow would
remain a set of stateless jobs with increasingly complex coordination.

## Pay for Qodo

Qodo was useful during its trial and would remove most maintenance. Its free
version is not producing reviews, however, and the custom workflow delivered
better observed value per dollar. Reconsider if maintenance cost reverses that
comparison.

## Adopt Gemini Enterprise Code Review

This restores a managed Gemini reviewer but adds enterprise billing,
configuration, and product coupling to replace a consumer service that was
withdrawn. It is a poor fit for a solo public repository.

## Keep the Stateless OpenRouter Reviewer

This retains broad model access and the best observed review economics with no
migration. It leaves the core problem intact: every qualifying push purchases
another review of mostly repeated context.

# Consequences

## Positive

* Review timing follows PR risk and readiness rather than commit count.
* Durable hunk and finding state removes redundant calls; prefix caching can
  further reduce the price and latency of repeated context.
* Serialisation, debounce, and idempotency prevent races and duplicate comments.
* Per-PR model, token, cache-hit, finding, and disposition data strengthen the
  existing evaluation flywheel.
* Recurring defects can feed back into tests, skills, repository instructions,
  and agent selection, reducing failures earlier in the development loop.
* A portable real-work corpus supports model evaluation that reflects actual
  tasks and creates a foundation for review agents in future products.
* Cloudflare-hosted open-weight models can improve economics while model and
  provider escape hatches preserve competition.

## Negative

* Webhook security, GitHub App permissions, retries, state migrations, model
  churn, and comment reconciliation become owned infrastructure.
* Incremental review can miss interactions outside changed hunks; risk rules and
  a final full-review path mitigate but cannot eliminate this.
* Durable Objects, Workflows, Workers AI, and AI Gateway pricing and limits can
  change. Spend limits and per-PR budgets are required before production use.
* Session affinity improves prefix-cache probability but is not durable storage
  or a cache guarantee; savings must be measured from returned usage data.
* The private benchmark is observational: task mix, prompts, and coding agents
  change over time, while dispositions are sparse and subjective. Results need
  stratification and periodic controlled replays before driving model choices.
* A growing corpus can retain source code, prompts, or future user-authored
  content longer than intended. Data minimisation, access control, retention,
  deletion, and redaction must be designed before collecting non-public data.
* Consolidating on Cloudflare reduces operational surface while increasing
  Cloudflare concentration risk.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt