Claude Code Review Criteria: Automate vs. Keep Human

Verified against Anthropic's Code Review, ultrareview, and security-review documentation as of July 2026. This surface moves fast — check pricing and availability before you budget against any number here.

Key takeaways

• Automate security fully, business logic partially (only what you write down), and leave architecture and maintainability to humans.

• REVIEW.md, not CLAUDE.md, is where review severity and scope actually get enforced — and it should stay under 100 lines.

• The check run's neutral conclusion doesn't mean you can't gate on it: a machine-readable severity payload lets you build your own merge block.

• Code Review and ultrareview are both unavailable under Zero Data Retention, which rules them out for a lot of regulated teams by default.

• An AI reviewer reading forked PRs is processing attacker-controlled text while holding credentials — treat it as an attack surface, not just a tool.

Introduction

The pull request is 400 lines. It touches payment reconciliation, adds two database queries, and changes how retries work on a webhook to a third-party provider. An agent wrote most of it in about twenty minutes.

The automated reviewer posts twenty-three inline comments.

Nineteen are formatting preferences, redundant null checks, a suggestion to extract a function used exactly once. Two are technically correct but describe code the PR never touched. One is a genuine improvement. The race condition in the retry handler — the one that will double-charge a customer roughly once a month under load — gets no mention at all.

The human reviewer, looking at 400 lines of plausible, well-formatted, confidently-commented code they didn't write and can't mentally trace, approves it. Not from laziness. From the specific fatigue of reviewing code that looks right, produced faster than anyone can actually read it.

Three sprints later someone adds a filter to collapse the bot's comments. The tool keeps running, keeps billing, and stops being read.

That's the state of code review on a lot of teams right now, and it has two separate causes that get treated as one problem. The machine is reviewing the wrong things. The human has stopped reviewing the right ones. Fix either alone and nothing improves.

This piece is a framework for fixing both: how to split review work between machine and human, and — with actual configuration, not hand-waving — how to encode the machine's half so it catches the race condition instead of the null check. Along the way it covers what almost nothing else written on this topic does: how to build a merge gate on a reviewer Anthropic explicitly says can't block one, what reviews actually cost once your three free runs are spent, and why an AI reviewer wired into CI is an attack surface with a working exploit already published against it. There's a complete, copy-paste REVIEW.md further down — if you skim, that's the part to stop for.

Why AI-Generated Code Broke Code Review

Traditional review rests on an assumption that no longer holds: the author understood the system.

When a colleague submits a PR, review is a check on execution. You assume they knew the retry utility already existed, knew the reporting module deliberately allows raw SQL, knew the legacy adapter's loose typing is tracked debt rather than an oversight. You're looking for mistakes inside a shared understanding.

An agent has no such understanding. It produces plausible solutions rather than reasoning about the whole system, and even with project context loaded it will confidently ship code that compiles, passes tests, reads clean, and is wrong in ways that only surface later:

• business logic duplicated because it didn't know the utility existed

• an abstraction built for a single call site

• architecture that diverges from a pattern the team settled on eighteen months ago

• an edge case handled incorrectly but consistently, so the tests pass anyway

• a performance regression invisible until real traffic arrives

The reviewer's actual question has changed. It's no longer does this code work? It's is this the right solution for this codebase? — which requires knowledge the code's author never had.

Two forces make this worse, and both get underestimated.

Volume flipped the economics. Anthropic's own framing is blunt: code output per engineer at Anthropic grew roughly 200% in a year, and review became the bottleneck. Writing code got fast. Reviewing it didn't. Teams reached for an automated reviewer to fix a problem automation itself created, then configured that reviewer as if capacity were still cheap. If you're still working out how AI fits at the front of that pipeline rather than the back, our guide to a plan-first AI coding workflow covers the upstream half — the part that determines how reviewable the output is in the first place.

Trust doesn't come back once spent. A reviewer right nine times in ten gets read. A reviewer right one time in ten gets muted, and stays muted after you fix it. Anthropic's own open-source review plugin bakes this in at the prompt level: sub-agents are told not to flag anything they're not certain about, because false positives cost more than they're worth. Each candidate finding gets scored 0–100 and anything under 80 is dropped by default. The managed Code Review service goes a step further, running a separate verification pass that checks each candidate against actual code behavior before it's allowed to surface as a comment.

The default configuration of any AI reviewer isn't neutral — it's a guess, averaged across every codebase the model has seen. The work here is replacing that guess with a specification.

The Code Review Framework: Five Layers to Triage

A complete code review covers five layers. None of this is new or AI-specific — it's what good reviewers have always done, roughly in this order:

1. Architecture — does this fit the system we have?

2. Business logic — does it preserve what the business actually requires?

3. Security — does it expose anything?

4. Performance — does it hold up under real traffic?

5. Maintainability — can someone change this in a year?

What's new is that an agent can now do part of this work, and the quality of your review process comes down almost entirely to triaging those five layers correctly between machine and human. Hand the machine a layer it can't actually do and you get confident nonsense. Keep a layer the machine could own and you're burning scarce human attention on work that should be free.

Here's the triage that holds up in practice:

Two consequences follow, and both cut against most teams' instincts.

Your automated reviewer should say much less than it's capable of saying. Its default output spans all five layers at roughly equal weight. The right configuration deletes most of layers 4 and 5 and puts nearly everything into 2 and 3.

Human review doesn't shrink. It relocates. Reviewers stop reading for syntax and start reading for fit — harder, slower, and considerably more valuable. Budget for that honestly and you get a team that ships faster. Skip it and you get a team that ships faster for two quarters, then spends a year paying for it.

Encoding the Machine's Half: How to Configure REVIEW.md

This is the step nearly every team skips, and it's where the leverage lives.

Claude Code Review reads two files from your repository, and they don't carry equal weight:

• CLAUDE.md is shared project context, used across every Claude Code task. The reviewer reads it and flags newly introduced violations — as nits. Useful, but that severity ceiling means a CLAUDE.md rule can't, by itself, be treated as a merge risk. It also works bidirectionally: if your PR makes a CLAUDE.md statement stale, Claude will tell you the docs need updating.

• REVIEW.md is review-only instruction at your repository root, injected into the system prompt of every agent in the review pipeline as its highest-priority block. This is where you change what gets flagged, at what severity, and how findings are reported.

If you take one thing from this article, take this: CLAUDE.md describes your codebase, REVIEW.md describes your review. Teams put review criteria in CLAUDE.md, watch every finding come back tagged as a nit, and conclude the reviewer has no sense of priority. It has exactly the sense of priority it was given.

One mechanical detail that trips people up: REVIEW.md gets pasted verbatim into the prompt. The @ import syntax that works in CLAUDE.md isn't expanded here, and referenced files aren't read in. If you want a rule enforced, the rule itself has to be in the file. Pointers do nothing.

Model severity around the merge decision

Reviews tag findings as Important (🔴 — fix before merge), Nit (🟡 — minor, non-blocking), and Pre-existing (🟣 — a bug this PR didn't introduce). That taxonomy is a starting point, not a policy. Redefine it in REVIEW.md against one binary question: does this finding change whether the PR should merge?

• Important — data loss, security exposure, incorrect money, broken auth, a migration you can't roll back, PII in logs. Rare. Six bullets, maximum. If your Important list runs longer than that, it's a wish list, not a severity definition.

• Should-fix — a real defect, but not a merge risk. Reported, doesn't stop the PR.

• Nit — capped, numerically, with the remainder reported as a count rather than as individual comments. Anthropic's own example caps at five. Zero is defensible if your linter already owns that surface. Don't pay a language model to have opinions about formatting.

• Pre-existing — separated or suppressed. A reviewer that reports bugs the author didn't write is technically correct and practically corrosive: it makes the PR feel unmergeable for reasons that have nothing to do with the PR. There's a real counter-argument here, though — this is exactly the category where Code Review earns its keep. TrueNAS's documented win was a pre-existing type mismatch in adjacent code that was silently wiping an encryption key cache on every sync. Suppress the category by default, but consider surfacing it on the paths where correctness is expensive.

The four instructions almost nobody writes

Most REVIEW.md files in the wild are just a list of things to check. The higher-leverage instructions constrain how the reviewer speaks, and almost nothing published discusses them.

Set a verification bar. Force the reviewer to earn its assertions: "Behaviour claims require a file:line citation in the source. Do not infer behaviour from a function's name." Most false positives come from the reviewer pattern-matching on an identifier rather than reading the code. This one line kills a large share of them.

Define re-review convergence. This fixes the problem every team hits by week three: a one-line change reaches round seven because each re-review invents fresh nits. Write the rule down: "On any review after the first, post Important findings only. Suppress new nits entirely."

Use a graduated bar, not a binary skip. Skipping a path entirely is blunt. For code that deserves some scrutiny but not full scrutiny, raise the bar instead: "In scripts/, report only if near-certain and severe."

Shape the summary. Ask the review body to open with a tally — 2 important, 4 nits — and to lead with "no blocking issues" when that's true. Authors want the shape of the verdict before the detail. A reviewer that buries "this is fine" under four paragraphs of prose is a reviewer people stop opening.

A REVIEW.md worth copying

Exclusions first, severity second, repo-specific checks third, reporting discipline last.

# Review instructions

## Do not review

- `**/*.generated.ts`, `**/migrations/**`, `package-lock.json`, `**/__snapshots__/**`

- Anything CI already enforces: lint, formatting, type errors, spellcheck.

- Maintainability and readability opinions — owned by the human reviewer.

- In `scripts/`: report only if near-certain and severe.

## What Important means here

Reserve Important for findings that break behaviour, leak data, move money

incorrectly, or block a rollback:

- Any change under `src/billing/**` that can double-charge, under-charge, or

  fail to charge. Trace the retry and idempotency path explicitly.

- User input reaching a query without parameterisation.

- An endpoint added or modified without an authorisation check.

- A `logger` call that could emit PII (email, card, address, token).

- A migration that is not backward-compatible with the currently deployed version.

Everything else is Nit at most.

## Cap the nits

Report at most three Nits per review. If you found more, write "plus N similar

items" in the summary rather than posting them inline.

## Repo-specific

- Every new API route must have an integration test in `tests/api/`.

- Database access goes through `src/repositories/` — never in a controller.

- Database queries must be scoped to the caller's tenant.

- `src/legacy/adapters/**` is out of scope. Known debt, tracked, do not comment.

## Verification bar
- Behaviour claims require a `file:line` citation in the source. Do not infer

  behaviour from naming.

- State the concrete failure mode, not the principle. "This double-charges when

  the webhook retries inside the 30s window" — not "consider idempotency."

- You are given the PR title and description. Use them: intent matters.

## Re-review

- After the first review, post Important findings only. Suppress new nits.

## Summary

- Open with a tally: "2 important, 3 nits". If nothing is blocking, say so first.

Look at what that file is mostly made of. Not "check for bugs" — exclusions, reporting discipline, and things true only in this repository.

SQL injection, XSS, hardcoded secrets, unhandled rejections: the model already knows to look for these, and the dedicated security path covers them too. Restating them spends your criteria budget on work that was already free. The billing idempotency window, the tenant-scoping rule, the legacy-adapter carve-out — those aren't free, because no reviewer on earth could infer them from the code alone. That asymmetry is the entire job.

And keep it short. Anthropic says this outright: a long REVIEW.md dilutes the rules that matter most. Every line you add weakens every other line. Aim under a hundred lines and treat additions as a budget, not a backlog.

Scope: Most Noise Gets Generated Before Any Criterion Runs

Criteria can't save a reviewer pointed at the wrong code.

Set the diff boundary deliberately. The local /code-review command defaults to your branch's commits ahead of upstream plus uncommitted working-tree changes. That's right for a pre-push check and wrong for a merge decision. For a merge decision, pass an explicit ref range — /code-review main...my-feature reviews the committed diff the PR would actually contain, regardless of how the branch's upstream is configured. You can also target a single file, a branch, or a PR number, which is the right move when someone wants a second opinion on one contested module rather than the whole change.

Choose a trigger that matches how your team pushes. The managed Code Review GitHub App runs once on PR open, on every push, or only on request. Teams pushing in small increments who also run on-every-push end up paying for — and being asked to read — a review of every intermediate broken state.

Here's the trap, and it's a genuine one: @claude review is not a one-off command. It starts a review and subscribes the PR to push-triggered reviews from that point forward, in any mode. The single-shot version is @claude review once. A team running Manual mode specifically to control cost, who then types @claude review on a long-running PR, has just opted that PR into a review on every subsequent push without meaning to. once is also what you use to retry a failed or timed-out run — the Re-run button in GitHub's Checks tab does not retrigger Code Review.

The higher-signal default for most teams: review once on PR open, @claude review once for a deliberate re-read.

Keep PRs small — and this is a review-quality control now, not a style preference. An agent can produce 800 changed lines before lunch. Reviewers measurably spend less attention per line as diffs grow, and the automated reviewer degrades along the same curve: cost scales with diff size, and past a certain point ultrareview simply refuses the job. If AI is writing the code, the discipline of splitting work into single-objective PRs is doing more work than it ever did before.

Set effort against stakes, not habit. Lower effort levels return fewer, higher-confidence findings; high through max widen coverage and start surfacing uncertain findings alongside confident ones. A config change doesn't need max. The payment path does.

How to Build a Claude Code Review Merge Gate on a Reviewer That Officially Can't Block

The documentation is explicit: Code Review doesn't approve or block your PR. Findings post as inline comments, and the check run always completes with a neutral conclusion so it never trips branch protection. Every article on this topic stops there and concludes you simply can't gate on it.

That conclusion is wrong, and it's the most useful thing in this whole product that nobody writes about.

Every review writes a severity table into the Claude Code Review check run, and the last line of that check run's output is a machine-readable payload. You can read it from your own CI:

gh api repos/OWNER/REPO/check-runs/CHECK_RUN_ID \

  --jq '.output.text | split("bughunter-severity: ")[1] | split(" -->")[0] | fromjson'

That returns counts per severity — something like {"normal": 2, "nit": 1, "pre_existing": 0}, where normal is the Important count. A non-zero normal means Claude found at least one bug it considers worth fixing before merge.

Which means you can build the gate yourself, in your own required check, on your own terms:

• Advisory (the default): do nothing. Comments stay comments.

• Soft gate: fail your own check when normal > 0, but only on protected paths — src/billing/**, src/auth/**, migrations. Everywhere else, stay advisory.

• Hard gate: fail on any normal > 0, anywhere. Only sane once your precision audit (below) is clearing 80%.

This is the single highest-leverage thing most teams aren't doing. It turns a bot people learn to scroll past into a control with actual consequences — scoped to the paths where consequences are warranted. Start on one directory and watch how fast that directory's REVIEW.md gets tightened once a false positive costs someone a merge.

Two caveats that matter. Runs are best-effort: a failed run posts nothing and doesn't retry on its own, so your gate has to treat "no result" as pass, not fail, or a timeout turns into an outage. And don't build this until precision is actually good — a hard gate on a noisy reviewer isn't a quality control, it's a lesson in how to route around branch protection.

Claude Code Review Options: Five Mechanisms, Five Jobs

Teams routinely pick one of these and expect it to do all five jobs. It won't.

One version note that has broken people's scripts: the local command was called /simplify before v2.1.147. From v2.1.154, /simplify became a separate cleanup-only review that applies fixes without hunting for bugs. If you scripted /simplify expecting bug detection, switch to /code-review --fix.

The custom skill is the option most teams haven't reached for, and it's where the interesting work happens. It's just a markdown file with frontmatter — name, description, allowed tools, model — whose body is the review instruction. That's the entire mechanism.

The wrong use of a skill is "do a code review." The built-in reviewer already does that better, with an agent fleet and a verification pass you'd otherwise have to rebuild from nothing. The right use is the review your team already does by hand and dreads:

• Is this migration backward-compatible with the version currently deployed?

• Is every new user-facing string in the i18n catalogue?

• Does this API change break the mobile client on version N-1?

• Does this endpoint's response shape still match what the OpenAPI spec promises?

• Does this touch a field the data-retention policy says we must not persist?

Narrow, repeatable, expensive to do by hand, cheap to specify. That last one isn't hypothetical — in a regulated environment it's the difference between a routine merge and a compliance incident. If your review surface spans cross-system contracts rather than a single codebase, this overlaps with the broader integration risk covered in our piece on enterprise API integration in Australia.

The rule: generic check, use the built-in reviewer and tune it with REVIEW.md. Check that's yours alone, write a skill.

Five Ways Teams Break Claude Code Review

Every failure mode below is one I've watched a competent team walk into. None of them are stupid mistakes — each is a reasonable default that turns out to be wrong once you've run the tool on a real codebase for a month.

Putting review criteria in CLAUDE.md. The single most common one. The reviewer reads CLAUDE.md, so the rules appear to work — but everything it finds comes back tagged as a nit, because that's the only severity CLAUDE.md violations get. The team concludes the tool has no judgment. It has exactly the judgment its config gave it, in the wrong file. Criteria that need to block go in REVIEW.md.

Running on every push. Reasonable-sounding: catch problems early. In practice you pay for — and are asked to read — a review of every intermediate broken state, at 3–5x the cost of reviewing once, for a quality gain close to zero. Worse, @claude review silently subscribes the PR to this mode even for teams that thought they were in Manual. Review once on PR open; re-read deliberately with @claude review once.

Gating before precision is good. A team sees the severity payload, gets excited, and wires a hard merge block to it in week one. The reviewer's precision is still 50%, so half the blocks are false, and within a sprint the engineers have learned to bypass branch protection — a habit that outlasts the reviewer. Earn the gate. Advisory until precision clears 80%, then soft-gate one protected directory, then widen.

Reading the bot's silence as approval. The check run completes neutral whether the reviewer found nothing or simply didn't run — best-effort means a timeout posts no comment at all. A green-looking PR is not a reviewed PR. If your process treats "no comments" as "Claude approved," a failed run becomes a silent skip on exactly the change nobody else looked at either.

Never revisiting the criteria. REVIEW.md gets written once, during setup, and then rots. The codebase moves, the reviewer keeps flagging the deliberate loose typing in the legacy adapter, engineers start muting it, and six weeks later nobody reads it. Criteria are a living document. Every dismissed finding is a missing line, and the file only converges if someone adds those lines.

The pattern underneath all five: teams treat the reviewer as a product to install rather than a policy to maintain. The install takes an afternoon. The policy is the job.

Zero Data Retention and Regulated Industries: The Blocker Nobody Mentions

This is the section that should sit at the top of every article on this topic. It sits at the top of none of them.

The managed Code Review service is not available to organisations with Zero Data Retention enabled. Ultrareview is unavailable under ZDR too, and also unavailable when you run Claude Code through Amazon Bedrock, Google Cloud's Agent Platform, or Microsoft Foundry — it executes on Anthropic's own web infrastructure and needs Claude.ai account authentication, not an API key.

Read that again if you work in financial services, healthcare, or government. The organisations with the strongest case for a tireless reviewer on every PR — regulated, audited, high-consequence — are frequently the ones locked out of the managed product, because the data-residency and retention posture they're contractually required to hold is exactly the posture the product doesn't support.

That's not a reason to give up on AI code review. It's a reason to build it where your data already lives — which is exactly the shape of problem our AI software development work in regulated industries is built around.

That's the path we took with Padua Solutions, a fintech platform doing Australian paraplanning and Statement of Advice work under financial-services compliance obligations. The managed service wasn't an option, so we built the review pipeline into their own GitLab CI/CD, running on Amazon Bedrock:

• Three specialised agents in parallel on every non-draft merge request — code quality, security, and API compliance — with a mixed-model strategy: Opus for security, where reasoning depth matters most for catching real vulnerabilities, Sonnet for the pattern-based checks.

• Advisory, not blocking. Bedrock throttling, errors, or malformed responses degrade gracefully rather than stopping merges — the same best-effort principle the managed service uses, and the same reason your own gate needs to fail open.

• Secrets never reach the model. Sensitive files (.env, private keys, credentials) are filtered before any diff leaves the runner. A secret-pattern scanner aborts the review outright if anything looks like a credential, and the output is scanned again on the way back, stripping residual secrets and any GitLab quick-action commands the model might have been induced to emit.

That last control isn't paranoia — it's the mitigation for the attack described in the next section, and it's why the pipeline treated the model as an untrusted component from day one, the same posture we push in our guide to integrating security into DevOps.

Claude Code Review Security Risks: Your Reviewer Is an Attack Surface

One section on this, because it gets skipped and the consequence is severe.

An automated reviewer in CI reads a diff and acts on what it reads. Trigger your workflow on pull requests from forks and untrusted text — code comments, hello@whitefox.cloud commit messages, test fixtures, the PR body — enters an agent's context directly. That agent may hold credentials, repository write access, or the ability to comment as a trusted identity.

This is prompt injection with a supply-chain payload attached, and it's already happened.

In June 2026, Microsoft Threat Intelligence published a disclosure against Anthropic's Claude Code GitHub Action. Untrusted GitHub content — issue bodies, PR descriptions, comments — could induce the agent to read the CI runner's environment and exfiltrate workflow secrets, including the ANTHROPIC_API_KEY itself.

Two details from that write-up matter well beyond this one action:

• The safety layer was laundered, not broken. The model's refusal training almost certainly catches "print this API key" — a string starting with sk-ant- is an obvious trigger. The exploit reframed the request as a compliance review and had the model truncate the first several characters before emitting the rest. The refusal never fired. GitHub's secret scanner didn't match the mangled string either.

• The sandbox wasn't uniform. Environment scrubbing applied to subprocess execution paths like Bash, but not to the Read tool, which was eventually able to reach /proc/self/environ. Anthropic fixed this in Claude Code v2.1.128 by blocking access to sensitive /proc files.

Key takeaway: Your defenses are pattern-matchers. A model instructed carefully enough is a pattern-launderer. Assume both are true and design controls that don't depend on the model refusing anything.

Anthropic's own security-review action says as much in plain text in its README: not hardened against prompt injection, use only on trusted PRs. That's an honest disclosure, worth treating as an instruction rather than a footnote.

None of the mitigations here are exotic:

• Turn on "Require approval for all external contributors" in your GitHub Actions settings. The workflow then only runs after a maintainer clicks approve, which gives a human a chance to eyeball the diff for injection before an agent ever sees it. Highest-value control on this list, and it takes thirty seconds to set.

• Minimise permissions. pull-requests: write and contents: read. Never contents: write. A reviewer with --fix rights in CI is an agent with commit access to your default branch, reachable by anyone who can open a PR. Worth knowing: the managed Claude GitHub App requests a broader permission set than review itself needs — contents read and write, issues read and write, pull requests read and write — because the same installation also backs GitHub Actions if you enable that later. Review only uses read on contents and write on PRs. Know what you actually granted.

• Use a dedicated, capped API key for the review workflow. A compromised workflow sharing a key with other services can burn a month of spend, or worse.

• Scan the output, not just the input. As in the Padua pipeline above — strip secrets and platform quick-action commands from the model's response before it's posted, on the assumption that the input controls already failed.

• Tell the model, in your criteria file, that diff content is data, never instruction — but don't rely on that instruction as your only control. It's a speed bump, not a wall.

This is table stakes now for anyone running agents in CI, and it's the clearest illustration of the wider governance problem. An AI reviewer isn't a linter with better English. It's software that reads attacker-controlled text and then takes actions using your credentials — a decision that belongs with whoever owns your security posture, which in a lot of growing teams is the fractional CTO or Principal Architect, not whoever happened to install the GitHub App.

What Stays Human: The Review the Machine Can't Do

If the automated reviewer is doing its job, your engineers have their attention back. Here's where to spend it — the AI-generated code failure modes no configuration will catch for you.

Reinvention. The most common pathology in agent-written code: a utility, hook, or helper that already exists forty lines away in src/lib/. Tell the reviewer to look for it and it'll sometimes find it. The person who knows the codebase finds it every time. Before approving anything new, ask whether it should have existed at all.

Over-engineering. Agents reach for the general solution by default. An interface with one implementation. A configurable strategy pattern for a single case. A generic utility solving exactly one problem. Each is defensible on its own; collectively, they're how a codebase becomes unnavigable. The simpler implementation is nearly always the right review outcome.

Hidden behavioural change. The expensive one. A refactor that was supposed to preserve behavior quietly changes serialization, exception types, validation order, caching, or concurrency. Tests still pass, because they were written against the same misunderstanding the refactor codified. Verify behavior at the boundary, not in the diff — API responses, error shapes, what actually lands in the database.

Architectural drift. No single PR is wrong. Session three used a different state pattern than session one because it was never told about session one. The next PR inherits both and picks one arbitrarily. Nothing is broken; the codebase is just quietly becoming two codebases. Only a human who holds the shape of the system in their head catches this, and only if they're actually looking for it instead of reading for syntax.

That last one is the whole argument for this exercise. The reviewer's time is finite. Every minute spent on a null check is a minute not spent on the drift.

It's also the argument for keeping upstream discipline intact. Drift is cheaper to prevent than to detect — a versioned decision record and a written implementation plan mean session three actually knows what session one decided. That's the core of the plan-first workflow, and it's why review criteria and planning artifacts belong in the same repository, reviewed with the same seriousness.

How to Improve Claude Code Review Accuracy Over Time

Criteria aren't configuration you set once. They're a document maintained against evidence, and almost nobody does this.

Use the built-in signal. Every Code Review comment arrives with 👍 and 👎 already attached. Anthropic collects reaction counts after the PR merges and uses them to tune the reviewer. It costs a click — tell your team to use it. It's worth knowing what reacting doesn't do: it doesn't re-run the review, and replying to an inline comment doesn't prompt Claude to respond. To act on a finding, fix the code and push.

Run a precision audit at four weeks. Pull the last thirty reviews. Classify every comment: acted on, wrong, or irrelevant. Precision — the share of findings that were real and relevant — is the number that decides whether engineers keep reading. Below roughly 60%, they stop, and at that point recall doesn't matter because nobody's looking anymore.

Every dismissed finding is a missing line in REVIEW.md. This is the actual feedback loop. If the reviewer keeps flagging the deliberate loose typing in the legacy adapter, that's not a defect in the reviewer — it's an unwritten rule in your codebase. Write it down. Over two or three cycles the noise floor drops sharply, and permanently, because the rule now lives with the code.

Track the right number. "PRs reviewed" only tells you the tool is running. What matters is already on the analytics dashboard: comments auto-resolved because a developer actually addressed the issue. If that number stays flat while spend climbs, the reviewer is talking to an empty room. Anthropic's own internal benchmark: after deploying Code Review, the share of PRs receiving substantive review comments went from 16% to 54%, with engineers marking fewer than 1% of findings as incorrect. That's the bar. If you're nowhere close, the problem is your criteria, not the model.

Claude Code Review Pricing: What Reviews Actually Cost

Cost is almost entirely absent from public writing on this topic, and it's the first question any engineering manager actually asks.

The documented shape as of mid-2026:

Two corrections to what's floating around elsewhere:

• The free runs are a one-time allotment, not a trial window. Anthropic's docs are explicit that Pro and Max subscribers get three free ultrareview runs per account, that they don't refresh monthly, and that once they're gone every run bills to usage credits. A stopped or failed run still counts against the three. Anthropic hasn't published a fixed calendar date for when the offer itself closes — the current docs describe it as ending "after the free run period ends," without a date — so budget for the runs disappearing per-account rather than assuming a specific cutoff. You also need usage credits enabled before you can launch a paid run at all; Claude Code blocks the attempt and points you to billing settings if you haven't turned that on.

• The $15–$25 average is an average, and the tail runs long. Practitioners have published measurements of a single review costing $31.54 on a 70-line Markdown diff, and $77.96 on a PR in a larger Go codebase. Anthropic's own figure scales with PR size, codebase complexity, and how many findings need verification — and "codebase complexity" is doing a lot of quiet work in that sentence.

Key takeaway: Budget off the tail, not the average. A monorepo doesn't behave like a 70-line diff, and the invoice will say so.

Run that against a team merging forty PRs a week and the reviewer becomes a real line item — plausibly comparable to a headcount fraction. Which makes scope and criteria cost controls, not just quality controls:

• Every-push triggering multiplies spend by your push frequency — typically 3–5x, for a marginal quality gain. And remember @claude review silently enrolls a PR into that mode.

• Excluding generated files removes tokens from the review, not just comments from the PR.

• Effort level is a direct cost dial. Reserve max, and ultrareview, for paths where a defect is genuinely expensive. The rule is simple: run the deep review where a production bug would cost more than the review itself. Short list. Real one.

• Set the monthly spend cap before rollout, not after the first invoice. Code Review supports an organisation-level cap in admin settings. When it's hit, Code Review posts a single comment explaining the review was skipped and resumes next billing period — a far better failure mode than a surprise bill, but only if someone chose the number on purpose.

• Watch the per-repo average cost column. Within a week it'll tell you which repository is quietly eating the budget.

Teams that abandon these tools over cost usually aren't actually paying for review. They're paying for reviews of lockfiles, on every push, at maximum effort.

Here's the version that survives a budget conversation. A team of eight merging forty PRs a week, reviewing once on PR open at the default effort, at the documented $15–$25 average, lands somewhere around $2,600–$4,300 a month. Turn on every-push triggering and stop excluding generated files and that number can triple without anyone deciding it should. Against it: the reviewer exists to catch the class of defect — the double-charge, the missing auth check, the non-reversible migration — where a single production incident costs more in engineer-hours and customer trust than a quarter of review spend. The tool isn't competing with your linter, which is free and which you already half-ignore. It's competing with the cost of the bug it's configured to find. Frame the line item that way and it defends itself; frame it as "AI review, $4k/month" and it dies in the first cost review.

Best Practices

• Start by deleting, not adding. Your first REVIEW.md should be mostly exclusions. Get volume down, then tune for signal.

• Write criteria as failure modes, not principles. "Consider thread safety" produces nothing. "Two concurrent webhook deliveries inside the retry window must not both write to payments" produces a finding.

• Cap nits numerically, and report the remainder as a count. Reviewers don't self-limit unless told to.

• Set a verification bar. Require a file:line citation for behavior claims. Cheapest false-positive reduction available.

• Define re-review behavior. Important-only after round one, or a one-line fix reaches round seven on style alone.

• Don't restate what the model already knows. SQL injection is covered. Your billing invariants aren't.

• Keep REVIEW.md short. Length dilutes. Under a hundred lines. Every addition weakens every other line.

• Give the reviewer the PR title and description. Intent is context — a reviewer that knows a guard was removed deliberately won't flag its absence as a bug.

• Keep AI-generated PRs to a single objective. Diff size degrades machine review, human review, and your invoice at the same time.

• Version-control the criteria file and review changes to it. It's a policy document. Give it the scrutiny you give the code.

• Measure precision before you tune recall. A reviewer nobody reads has a recall of zero, whatever it found.

• Fail open, always. Best-effort runs mean "no result" must never mean "blocked."

Actionable Checklist

Run this on one repository. About ninety minutes.

Triage

• Decide, explicitly, which of the five layers your reviewer owns. Write it down. Security: yes. Maintainability: no.

Configure

• Create REVIEW.md at the repo root. Don't put review criteria in CLAUDE.md and expect it to carry weight — it arrives as nits.

• List every path that should never be reviewed: generated code, lockfiles, snapshots, vendored deps, ORM migrations.

• Write your Important definition. Six bullets, maximum.

• Write three to five repo-specific checks no external reviewer could infer. This is the entire value of the exercise.

• Cap nits at a specific number. Write the number.

• Add a verification-bar line and a re-review-convergence line. Two sentences, disproportionate payoff.

• Suppress or separate pre-existing findings — but consider allowing them on your highest-consequence paths.

• Keep the whole file under a hundred lines.

Scope and cost

• Choose a trigger: once on PR open, unless you have a specific reason to review every push.

• Tell the team: @claude review once, not @claude review, unless they actually mean to subscribe the PR.

• Set effort per code path, not globally. Reserve ultrareview for changes where a bug costs more than the run.

• Set the monthly spend cap before the first full week of use.

• Sanity-check the worst case: what does your largest repo's average review actually cost?

Security

• Enable "Require approval for all external contributors" on any fork-triggered review workflow.

• Audit workflow permissions. contents: read, pull-requests: write. Nothing more.

• Confirm your Claude Code version is at or above v2.1.128 if you run the GitHub Action.

• Scan the reviewer's output for secrets and platform commands, not just its input.

Gate

• Parse the check run's severity payload in your own CI. Start advisory.

• Once precision clears 80%, soft-gate on normal > 0 for protected paths only.

• Confirm the gate fails open when a review times out.

Keep it honest

• Book a thirty-review precision audit for four weeks out. Put it on someone's calendar now.

• After the audit, convert every dismissed finding into a line in REVIEW.md.

Conclusion

An AI reviewer without criteria is a well-read stranger with no stake in your codebase and no knowledge of your decisions. It tells you true things that don't matter and stays silent about the one that does — not from incapacity, but because nobody told it what mattered here.

The goal was never to review more code. It's to review the right layers with the right reviewer. Give the machine security and your written invariants, where it's tireless and consistent and genuinely better than a human at 5pm. Keep architecture, maintainability, and the hunt for behavioral drift with the people who hold the system in their heads — and give them back the time to actually do it.

A good REVIEW.md is under a hundred lines and takes an afternoon to write. The severity gate is twenty lines of CI. The audit loop that keeps it honest is an hour a month. Against that: a reviewer that catches the race condition in the retry handler before it double-charges a customer, and a team that's still reading its comments six months from now.

The model isn't the variable. The criteria are.

We've done this under real compliance constraints, not in slide decks. Our work with Padua Solutions embedded Claude across an entire regulated operating model — from a paraplanner's daily workflow to an auditable, secret-scrubbed review pipeline on Bedrock — for a team the managed GitHub App couldn't serve. Everything in this article — the REVIEW.md design, the severity gate, the CI hardening — is the kind of work we do as part of our Claude Code skills consulting for code review automation and engineering productivity, including as a Fractional CTO / Principal Architect engagement when you need hands-on leadership rather than advice.

Related reading on where AI fits in a delivery process: the plan-first AI coding workflow that makes output reviewable in the first place, integrating security into DevOps for the posture an agent in CI demands, and dedicated teams and fractional CTO leadership on the operating discipline behind it. Or reach us directly at hello@whitefox.cloud — send one repository, your REVIEW.md, and thirty recent reviews, and we'll tell you honestly whether the tool is earning its invoice.