Claude Code Review Criteria: Automate vs. Keep Human

Part 1 of a 3-part series on Claude Code Review.  Part 2 · Part 3

Verified against Anthropic's Code Review, ultrareview, and security-review documentation as of July 2026. This surface moves fast — check pricing and availability before you budget against any number here.

Key takeaways

• Automate security fully, business logic partially (only what you write down), and leave architecture and maintainability to humans.

• REVIEW.md, not CLAUDE.md, is where review severity and scope actually get enforced — and it should stay under 100 lines.

• Four instructions almost nobody writes — a verification bar, re-review convergence, a graduated skip bar, and summary shape — remove most of the noise a default configuration produces.

• There's a complete, copy-paste REVIEW.md further down. If you skim, that's the part to stop for.

Introduction

The pull request is 400 lines. It touches payment reconciliation, adds two database queries, and changes how retries work on a webhook to a third-party provider. An agent wrote most of it in about twenty minutes.

The automated reviewer posts twenty-three inline comments.

Nineteen are formatting preferences, redundant null checks, a suggestion to extract a function used exactly once. Two are technically correct but describe code the PR never touched. One is a genuine improvement. The race condition in the retry handler — the one that will double-charge a customer roughly once a month under load — gets no mention at all.

The human reviewer, looking at 400 lines of plausible, well-formatted, confidently-commented code they didn't write and can't mentally trace, approves it. Not from laziness. From the specific fatigue of reviewing code that looks right, produced faster than anyone can actually read it.

Three sprints later someone adds a filter to collapse the bot's comments. The tool keeps running, keeps billing, and stops being read.

That's the state of code review on a lot of teams right now, and it has two separate causes that get treated as one problem. The machine is reviewing the wrong things. The human has stopped reviewing the right ones. Fix either alone and nothing improves.

This part covers the first half: which of the five review layers to hand to the machine, and how to encode that decision in REVIEW.md so it actually holds. Part 2 covers turning findings into a real merge gate, choosing between the five delivery mechanisms, and what this costs. Part 3 covers the security risk of running an agent in CI and the constraints that rule the managed product out entirely for a lot of regulated teams.

Why AI-Generated Code Broke Code Review

Traditional review rests on an assumption that no longer holds: the author understood the system.

When a colleague submits a PR, review is a check on execution. You assume they knew the retry utility already existed, knew the reporting module deliberately allows raw SQL, knew the legacy adapter's loose typing is tracked debt rather than an oversight. You're looking for mistakes inside a shared understanding.

An agent has no such understanding. It produces plausible solutions rather than reasoning about the whole system, and even with project context loaded it will confidently ship code that compiles, passes tests, reads clean, and is wrong in ways that only surface later:

• Business logic duplicated because it didn't know the utility existed

• An abstraction built for a single call site

• Architecture that diverges from a pattern the team settled on eighteen months ago

• An edge case handled incorrectly but consistently, so the tests pass anyway

• A performance regression invisible until real traffic arrives

The reviewer's actual question has changed. It's no longer does this code work? It's is this the right solution for this codebase? — which requires knowledge the code's author never had.

Two forces make this worse, and both get underestimated.

Volume flipped the economics. Anthropic's own framing is blunt: code output per engineer at Anthropic grew roughly 200% in a year, and review became the bottleneck. Writing code got fast. Reviewing it didn't. Teams reached for an automated reviewer to fix a problem automation itself created, then configured that reviewer as if capacity were still cheap. If you're still working out how AI fits at the front of that pipeline rather than the back, our guide to a plan-first AI coding workflow covers the upstream half — the part that determines how reviewable the output is in the first place.

Trust doesn't come back once spent. A reviewer right nine times in ten gets read. A reviewer right one time in ten gets muted, and stays muted after you fix it. Anthropic's own open-source review plugin bakes this in at the prompt level: sub-agents are told not to flag anything they're not certain about, because false positives cost more than they're worth. Each candidate finding gets scored 0–100 and anything under 80 is dropped by default. The managed Code Review service goes a step further, running a separate verification pass that checks each candidate against actual code behavior before it's allowed to surface as a comment.

The default configuration of any AI reviewer isn't neutral — it's a guess, averaged across every codebase the model has seen. The work here is replacing that guess with a specification.

The Code Review Framework: Five Layers to Triage

A complete code review covers five layers. None of this is new or AI-specific — it's what good reviewers have always done, roughly in this order:

1. Architecture — does this fit the system we have?

2. Business logic — does it preserve what the business actually requires?

3. Security — does it expose anything?

4. Performance — does it hold up under real traffic?

5. Maintainability — can someone change this in a year?

What's new is that an agent can now do part of this work, and the quality of your review process comes down almost entirely to triaging those five layers correctly between machine and human. Hand the machine a layer it can't actually do and you get confident nonsense. Keep a layer the machine could own and you're burning scarce human attention on work that should be free.

Here's the triage that holds up in practice:

Two consequences follow, and both cut against most teams' instincts.

Your automated reviewer should say much less than it's capable of saying. Its default output spans all five layers at roughly equal weight. The right configuration deletes most of layers 4 and 5 and puts nearly everything into 2 and 3.

Human review doesn't shrink. It relocates. Reviewers stop reading for syntax and start reading for fit — harder, slower, and considerably more valuable. Budget for that honestly and you get a team that ships faster. Skip it and you get a team that ships faster for two quarters, then spends a year paying for it.

Encoding the Machine's Half: How to Configure REVIEW.md

This is the step nearly every team skips, and it's where the leverage lives.

Claude Code Review reads two files from your repository, and they don't carry equal weight:

• CLAUDE.md is shared project context, used across every Claude Code task. The reviewer reads it and flags newly introduced violations — as nits. Useful, but that severity ceiling means a CLAUDE.md rule can't, by itself, be treated as a merge risk. It also works bidirectionally: if your PR makes a CLAUDE.md statement stale, Claude will tell you the docs need updating.

• REVIEW.md is review-only instruction at your repository root, injected into the system prompt of every agent in the review pipeline as its highest-priority block. This is where you change what gets flagged, at what severity, and how findings are reported.

If you take one thing from this article, take this: CLAUDE.md describes your codebase, REVIEW.md describes your review. Teams put review criteria in CLAUDE.md, watch every finding come back tagged as a nit, and conclude the reviewer has no sense of priority. It has exactly the sense of priority it was given.

One mechanical detail that trips people up: REVIEW.md gets pasted verbatim into the prompt. The @ import syntax that works in CLAUDE.md isn't expanded here, and referenced files aren't read in. If you want a rule enforced, the rule itself has to be in the file. Pointers do nothing.

Model severity around the merge decision

Reviews tag findings as Important (🔴 — fix before merge), Nit (🟡 — minor, non-blocking), and Pre-existing (🟣 — a bug this PR didn't introduce). That taxonomy is a starting point, not a policy. Redefine it in REVIEW.md against one binary question: does this finding change whether the PR should merge?

• Important — data loss, security exposure, incorrect money, broken auth, a migration you can't roll back, PII in logs. Rare. Six bullets, maximum. If your Important list runs longer than that, it's a wish list, not a severity definition.

• Should-fix — a real defect, but not a merge risk. Reported, doesn't stop the PR.

• Nit — capped, numerically, with the remainder reported as a count rather than as individual comments. Anthropic's own example caps at five. Zero is defensible if your linter already owns that surface. Don't pay a language model to have opinions about formatting.

• Pre-existing — separated or suppressed. A reviewer that reports bugs the author didn't write is technically correct and practically corrosive: it makes the PR feel unmergeable for reasons that have nothing to do with the PR. There's a real counter-argument here, though — this is exactly the category where Code Review earns its keep. TrueNAS's documented win was a pre-existing type mismatch in adjacent code that was silently wiping an encryption key cache on every sync. Suppress the category by default, but consider surfacing it on the paths where correctness is expensive.

The four instructions almost nobody writes

Most REVIEW.md files in the wild are just a list of things to check. The higher-leverage instructions constrain how the reviewer speaks, and almost nothing published discusses them.

Set a verification bar. Force the reviewer to earn its assertions: "Behaviour claims require a file:line citation in the source. Do not infer behaviour from a function's name." Most false positives come from the reviewer pattern-matching on an identifier rather than reading the code. This one line kills a large share of them.

Define re-review convergence. This fixes the problem every team hits by week three: a one-line change reaches round seven because each re-review invents fresh nits. Write the rule down: "On any review after the first, post Important findings only. Suppress new nits entirely."

Use a graduated bar, not a binary skip. Skipping a path entirely is blunt. For code that deserves some scrutiny but not full scrutiny, raise the bar instead: "In scripts/, report only if near-certain and severe."

Shape the summary. Ask the review body to open with a tally — 2 important, 4 nits — and to lead with "no blocking issues" when that's true. Authors want the shape of the verdict before the detail. A reviewer that buries "this is fine" under four paragraphs of prose is a reviewer people stop opening.

A REVIEW.md worth copying

Exclusions first, severity second, repo-specific checks third, reporting discipline last.

# Review instructions

## Do not review

- `**/*.generated.ts`, `**/migrations/**`, `package-lock.json`, `**/__snapshots__/**`

- Anything CI already enforces: lint, formatting, type errors, spellcheck.

- Maintainability and readability opinions — owned by the human reviewer.

- In `scripts/`: report only if near-certain and severe.

## What Important means here

Reserve Important for findings that break behaviour, leak data, move money

incorrectly, or block a rollback:

- Any change under `src/billing/**` that can double-charge, under-charge, or

  fail to charge. Trace the retry and idempotency path explicitly.

- User input reaching a query without parameterisation.

- An endpoint added or modified without an authorisation check.

- A `logger` call that could emit PII (email, card, address, token).

- A migration that is not backward-compatible with the currently deployed version.

Everything else is Nit at most.

## Cap the nits

Report at most three Nits per review. If you found more, write "plus N similar

items" in the summary rather than posting them inline.

## Repo-specific

- Every new API route must have an integration test in `tests/api/`.

- Database access goes through `src/repositories/` — never in a controller.

- Database queries must be scoped to the caller's tenant.

- `src/legacy/adapters/**` is out of scope. Known debt, tracked, do not comment.

## Verification bar
- Behaviour claims require a `file:line` citation in the source. Do not infer

  behaviour from naming.

- State the concrete failure mode, not the principle. "This double-charges when

  the webhook retries inside the 30s window" — not "consider idempotency."

- You are given the PR title and description. Use them: intent matters.

## Re-review

- After the first review, post Important findings only. Suppress new nits.

## Summary

- Open with a tally: "2 important, 3 nits". If nothing is blocking, say so first.

Look at what that file is mostly made of. Not "check for bugs" — exclusions, reporting discipline, and things true only in this repository.

SQL injection, XSS, hardcoded secrets, unhandled rejections: the model already knows to look for these, and the dedicated security path covers them too. Restating them spends your criteria budget on work that was already free. The billing idempotency window, the tenant-scoping rule, the legacy-adapter carve-out — those aren't free, because no reviewer on earth could infer them from the code alone. That asymmetry is the entire job.

And keep it short. Anthropic says this outright: a long REVIEW.md dilutes the rules that matter most. Every line you add weakens every other line. Aim under a hundred lines and treat additions as a budget, not a backlog.

Scope: Most Noise Gets Generated Before Any Criterion Runs

Criteria can't save a reviewer pointed at the wrong code.

Set the diff boundary deliberately. The local /code-review command defaults to your branch's commits ahead of upstream plus uncommitted working-tree changes. That's right for a pre-push check and wrong for a merge decision. For a merge decision, pass an explicit ref range — /code-review main...my-feature reviews the committed diff the PR would actually contain, regardless of how the branch's upstream is configured. You can also target a single file, a branch, or a PR number, which is the right move when someone wants a second opinion on one contested module rather than the whole change.

Choose a trigger that matches how your team pushes. The managed Code Review GitHub App runs once on PR open, on every push, or only on request. Teams pushing in small increments who also run on-every-push end up paying for — and being asked to read — a review of every intermediate broken state.

Here's the trap, and it's a genuine one: @claude review is not a one-off command. It starts a review and subscribes the PR to push-triggered reviews from that point forward, in any mode. The single-shot version is @claude review once. A team running Manual mode specifically to control cost, who then types @claude review on a long-running PR, has just opted that PR into a review on every subsequent push without meaning to. once is also what you use to retry a failed or timed-out run — the Re-run button in GitHub's Checks tab does not retrigger Code Review.

The higher-signal default for most teams: review once on PR open, @claude review once for a deliberate re-read.

Keep PRs small — and this is a review-quality control now, not a style preference. An agent can produce 800 changed lines before lunch. Reviewers measurably spend less attention per line as diffs grow, and the automated reviewer degrades along the same curve: cost scales with diff size, and past a certain point ultrareview simply refuses the job. If AI is writing the code, the discipline of splitting work into single-objective PRs is doing more work than it ever did before.

Set effort against stakes, not habit. Lower effort levels return fewer, higher-confidence findings; high through max widen coverage and start surfacing uncertain findings alongside confident ones. A config change doesn't need max. The payment path does.

Best Practices

• Start by deleting, not adding. Your first REVIEW.md should be mostly exclusions. Get volume down, then tune for signal.

• Write criteria as failure modes, not principles. "Consider thread safety" produces nothing. "Two concurrent webhook deliveries inside the retry window must not both write to payments" produces a finding.

• Cap nits numerically, and report the remainder as a count. Reviewers don't self-limit unless told to.

• Set a verification bar. Require a file:line citation for behavior claims. Cheapest false-positive reduction available.

• Define re-review behavior. Important-only after round one, or a one-line fix reaches round seven on style alone.

• Don't restate what the model already knows. SQL injection is covered. Your billing invariants aren't.

• Keep REVIEW.md short. Length dilutes. Under a hundred lines. Every addition weakens every other line.

• Give the reviewer the PR title and description. Intent is context — a reviewer that knows a guard was removed deliberately won't flag its absence as a bug.

• Keep AI-generated PRs to a single objective. Diff size degrades machine review, human review, and your invoice at the same time.

• Version-control the criteria file and review changes to it. It's a policy document. Give it the scrutiny you give the code.

Checklist: Framework and REVIEW.md

Triage

• Decide, explicitly, which of the five layers your reviewer owns. Write it down. Security: yes. Maintainability: no.

Configure

• Create REVIEW.md at the repo root. Don't put review criteria in CLAUDE.md and expect it to carry weight — it arrives as nits.

• List every path that should never be reviewed: generated code, lockfiles, snapshots, vendored deps, ORM migrations.

• Write your Important definition. Six bullets, maximum.

• Write three to five repo-specific checks no external reviewer could infer. This is the entire value of the exercise.

• Cap nits at a specific number. Write the number.

• Add a verification-bar line and a re-review-convergence line. Two sentences, disproportionate payoff.

• Suppress or separate pre-existing findings — but consider allowing them on your highest-consequence paths.

• Keep the whole file under a hundred lines.

Continue the Series

With the criteria written, the next question is what happens once the reviewer finds something — and whether "the check run can't block a merge" is actually true. Part 2 covers building a real merge gate on top of it, which of the five delivery mechanisms fits your team, and what all of this actually costs.