# PDR 002: Put local review outcomes before benchmark scores

- HTML version: https://robbiepalmer.me/projects/agentic-code-review/pdrs/002-local-outcomes-before-benchmark-scores
- Project: Agentic Code Review
- Status: Accepted
- Opened: 2026-09-30
- Decided: 2026-09-30

## Context

The production scorecard records what happened after this reviewer commented
on real changes. Those outcomes reflect the repository, its risk policy, and
the owner's judgment. They can also overfit one codebase and one person's
preferences.

External review benchmarks broaden the languages, repositories, and defect
classes under test. Their labels answer a different question. A technically
correct reference comment is not automatically useful to this workflow, and a
generated finding that misses the reference set is not automatically noise.

## Decision

Use observed local outcomes as the primary evidence for changing review policy,
prompts, and model selection. Keep external benchmarks as separate challenge
sets that can expose generalisation regressions. Do not merge their labels with
local dispositions into one score or treat benchmark agreement as a model of
the owner's preferences.

A benchmark may block a change that causes a clear, predeclared regression. It
cannot promote a candidate that loses useful local findings, creates harmful
local churn, or increases owner attention. Report unmatched benchmark findings
as unadjudicated until a fixed blind sample has been reviewed.

Keep provenance visible. Separate human-seeded and model-seeded references,
flag model-family overlap, and compare candidates on the same pinned cohort,
runner, budget, and matching policy.

## Consequences

The scorecard can detect narrow overfitting without pretending that a public
dataset defines product value. Model changes need evidence from both the
workflow they serve and a reproducible external check.

The project must maintain separate reports and explain disagreements instead
of publishing one convenient ranking. External reference recall can understate
novel correct findings, while an incomplete reference set can make valid output
look noisy.

## Evaluation

The policy is working when model and prompt decisions cite local downstream
outcomes and show the external challenge result separately. A benchmark result
must be reproducible from its pinned source, and every promotion must account
for local useful discovery, owner attention, safety, cost, and coverage.

## Migration note

This record extracts the evidence policy that was mixed into ADR 004 on the
same date. ADR 004 retains the technical decision to import AACR-Bench through
the DVC evaluation pipeline. The migration changes the record boundary, not
the policy or its Accepted status.

## Related records

- Project: [Agentic Code Review](https://robbiepalmer.me/projects/agentic-code-review.md)
- Informed by ADR: [ADR 002: DuckDB for the AI Review Scorecard](https://robbiepalmer.me/projects/agentic-code-review/adrs/002-duckdb-ai-review-scorecard.md)
- Implemented by ADR: [ADR 004: Use AACR-Bench as a Secondary Challenge Set](https://robbiepalmer.me/projects/agentic-code-review/adrs/004-use-aacr-bench-as-a-secondary-challenge-set.md)

---

Markdown index of this site: https://robbiepalmer.me/llms.txt