
How the Claude Code team thinks about verification
How Claude Code combines end-to-end, property, and differential tests with isolated gates, incident-based verifier evaluation, and maintained review rules.
On the Claude Code team, roughly 90% of the code is written and reviewed by Claude. That number sounds like a claim about generation quality, but the more important consequence is organizational: when agents produce changes at this volume, a team cannot personally inspect every line and remain the bottleneck. It has to build a system that produces evidence.
This is the central argument in Addy Osmani's account of how the Claude Code team approaches verification. Tests are not a final checkbox added after the agent claims to be finished. They are the mechanism that makes delegation possible. Without a feedback loop, a developer can supervise one or two agents because every result still has to be checked by hand. With a trustworthy loop, dozens or even hundreds of agents can work in parallel while failures move back through a defined path.
Verification is not merely a quality activity. It is the control system that determines how much work can safely leave a human's immediate attention.
The article is useful because it avoids treating "AI testing" as one generic activity. It separates user-facing end-to-end tests, property tests, and differential tests; explains the operational rules used by the Claude Code team; and describes how the team evaluates its own verifiers. It also leaves room for a hard question: if agents generate both the implementation and the checks, what stops them from agreeing on the same mistake?
Verification creates the capacity to delegate
The economic case begins with an old software principle: generating an output is only half the job; checking that the output is correct is the other half. Coding agents have made both halves cheaper, but not equally. They can write code quickly, and they can also write tests, specifications, harnesses, and benchmark scripts quickly. Once those artifacts exist as software, they can run repeatedly without requiring a new model call for every judgment.
Anthropic reports one concrete internal result from structured file generation: using a separate grader to check Claude's output against a rubric improved task success by up to 10 points compared with a standard prompting loop. The largest gains appeared on the hardest tasks. The exact benchmark and denominator are not disclosed in the post, so the number should be treated as a directional vendor result rather than an independently reproduced effect size. Its design lesson is still clear: separating the producer from a grader with explicit criteria is more reliable than asking the same agent whether it believes its work is complete.
The second half of the loop is to feed failures back into the system. If an agent repeatedly fails a check that looks for personal data in logs, the team can add a hook or a stronger rule to CLAUDE.md. The next run begins with that lesson already present. This is how a one-off correction becomes an institutional control.
Not all tests are created equal
Before choosing a test type, the article identifies two failure modes that can quietly corrupt an agent workflow. First, flaky tests teach the agent to rerun a failure rather than repair the code. The behavior looks like persistence, but it is really a trained habit of ignoring evidence. Second, slow tests encourage agents to batch fewer checks, which makes causal diagnosis harder and increases the temptation to defer verification until the end.
For heavy checks, the useful pattern is to run them before committing and pushing, while the change is still cheap to revise. The following three test styles offer different kinds of evidence.
| Test type | What it proves | Best moment to use it | Main limitation |
|---|---|---|---|
| End-to-end | The complete user flow works in a running system | Before merging any user-facing change | Slower and more environment-sensitive than unit tests |
| Property-based | A general invariant survives thousands of generated inputs | Parsers, serializers, state transitions, rewrites | Requires a property that is genuinely correct and meaningful |
| Differential | The new system matches the old oracle except for approved changes | Migrations and rewrites where behavior must remain stable | Depends on an old system that is itself trustworthy |
End-to-end tests: let the agent use the product
If a team can invest seriously in only one test category, the article recommends end-to-end tests. They describe real user journeys, such as registering, adding an item to a cart, checking out, and reaching a confirmation page. Their value is not only that they cover more code. It is that they are difficult to fool: either the whole flow works or it does not.
Unit tests remain faster, less flaky, and easier to isolate. But they can all pass while the assembled feature is broken because both the implementation and the unit tests may share the same incomplete model of the task. End-to-end tests give the agent something closer to hands and eyes on the running product. A verification skill can boot the application and drive it through browser automation, the Chrome DevTools Protocol, mobile simulators, computer use, or an API harness.
A feature map makes this more reliable. If each user-visible capability is written down with its route, keyboard shortcut, selectors, expected state, and completion condition, the same map can serve as a navigation guide and as a grading rubric. The agent is no longer inventing success after the fact. It is checking the change against a description of what the product is supposed to do.
Property tests: attack the assumptions
Most example-based tests encode the cases the author happened to imagine. Property-based tests start from a stronger statement that should always hold, then use a library such as fast-check, Hypothesis, or proptest to generate thousands of attempts to break it. This is especially valuable for coding agents because the generated cases are not created with the same assumptions as the implementation.
Common properties include:
- Round trip: encoding and then decoding a value returns the original value.
- Invariant: a condition remains true at every valid state, such as an account balance never becoming negative.
- Idempotency: running a migration, normalization step, or retry twice has the same result as running it once.
- Equivalence: an optimized implementation produces the same result as a simple reference implementation.
Properties are particularly useful during rewrites. The original written specification may have drifted away from production behavior, while the new implementation is being judged against an outdated interpretation. A carefully chosen property can preserve the behavior that matters without freezing every incidental detail of the old implementation.
Differential tests: use the old system as an oracle
Differential testing is a natural fit for large migrations in which behavior should remain unchanged. The same input is sent to the old and new systems, and any difference becomes a candidate bug or an explicitly approved behavior change. Inputs can come from production traffic or from a generator designed to explore the space.
The difficult part is not running two systems. It is making the comparison meaningful. Teams need to normalize irrelevant differences such as whitespace, timestamps, ordering that is not semantically significant, and environment-specific fields. They also need a reviewed list of intentional changes. Without those controls, the test either produces an ocean of noise or hides a real regression by labeling it as expected.
How the Claude Code team operates its verification layer
The second half of the article is about process. The team's most important principle is simple: verification means running the change, not describing what should happen. An iOS change should be tried in a simulator. A backend change should be started and exercised locally. A web change should be driven in a browser. The agent should behave as much like a QA engineer as the environment allows.
The team encodes this behavior in a repository-level verify skill under .claude/skills/. The skill explains how to start the system, where the important flows live, which tools to use, and what evidence counts as success. It is expected to be fast, reliable, and maintained like any other critical test harness. The agent learns to invoke it before committing.
Failures, not every diff, decide where humans are needed
No organization can read every generated change. The Claude Code team uses failures as a routing signal. Claude runs the checks and attempts to repair what fails. If a check is still failing after three rounds, the agent stops and asks for human attention. This is a better use of human time than scanning every passing diff, because it routes attention to a concrete conflict between the artifact and the evidence.
The people who own a part of the code also own its checks. They refine the checks until they flag what the owner would flag. When a check pulls in a human reviewer, it must explain why the review is necessary. The reviewer should not have to reconstruct the alarm from the diff alone.
Reliability checks are triggered by the diff
The team maintains a 39-item reliability checklist. Each item has a trigger that can be detected from a diff, a corresponding check, and links to incidents or pull requests that motivated it. One example concerns session resume behavior: when session storage or message types change, the reviewer asks whether a new field will load safely, because older transcripts will not contain it.
Claude reads the diff and runs only the checks whose triggers match, usually between three and ten items. Instead of asking a reviewer to memorize every historical outage, the system preserves the operational lesson and retrieves it when relevant. This is a stronger model than a static checklist because the knowledge can grow without making every review equally expensive.
One reviewer should improve, not multiply
When review quality is poor, the instinct is often to add another reviewer. The Claude Code team argues that tuning the primary reviewer is usually more effective than stacking a second, more expensive reviewer on top. A repository's REVIEW.md tells the reviewer what matters in that codebase: conventions, high-risk areas, known failure shapes, paths to skip, and the kind of evidence required.
Review rounds should build on one another. Later passes receive context from earlier findings and past failures instead of starting from zero. This reduces repeated analysis and helps the reviewer converge on unresolved issues rather than rediscovering the same surface.
The change cannot modify the check that judges it
This is one of the strongest controls in the article. If a change can edit the test, policy, or reviewer instructions that decide whether it passes, the gate has no independent authority. Code Review reads REVIEW.md from the base or default branch, never from the pull request. Rules that govern a change must live somewhere the change itself cannot alter.
The team also binds review results to specific commits. If new commits arrive while review is running, a result for the old commit is no longer evidence for the current code. Cheap checks should run first. If an agent pushes before running local tests, it has spent shared CI and review capacity on a defect it could have caught itself. Hooks can enforce a test run when the agent finishes, and a goal can require a passing suite before the work is considered complete.
New gates begin in observe mode
The Claude Code team stamps commits with the verification they received, including a deliberate none value so unverified commits remain queryable. They also built a commit gate that can block pushes or commits when the session did not include verification. The gate has three modes: off, observe, and enforce.
- Observe: log what would have been blocked, then inspect false positives before affecting developers.
- Enforce: block the action unless the commit explains why verification was not possible.
- Escape hatch: after two blocked attempts in a session, allow the next attempt so an imperfect agent does not become permanently stuck.
This staged rollout is important. A verification gate is itself software, and its false positives have a cost. Observe mode creates evidence for tuning the policy before the policy starts interrupting work.
Evaluate the verifier, not only the code
Once verification becomes a system, it needs its own quality metrics. The Claude Code team tracks how many merged changes are reverted, how many changes maintainers reject before merge, and a size-matched control set of changes that stayed clean. This prevents the team from celebrating fewer comments or faster reviews while the underlying defect rate is rising.
Anthropic's code review team has also built a formal evaluation from past incidents and reverts. For each case, the repository is reconstructed at the state in which the review originally happened, and a one-sentence answer key identifies the bug. Only findings that correctly describe that bug receive credit. Stable merged changes act as controls, which makes it possible to detect a reviewer that improves recall simply by flagging everything.
The evaluation separates three stages:
- Whether the raw search produced the relevant finding.
- Whether the verification stage kept a true finding.
- Whether the final review surfaced it to the user.
Reporting only one end-to-end score hides where the loss occurs. A reviewer may find a bug, lose it during verification, and appear no better than a reviewer that never found it. Stage-specific recall makes the failure diagnosable.
Validity is not the only dimension. A technically correct observation may be irrelevant. The team labels findings as Important, Nit, or Pre-existing. The last label matters because it prevents an old defect from being presented as something the current author introduced. The team tunes recall only up to the point where additional improvements would overfit the evaluation instead of catching useful bugs in production.
Turn misses into maintained rules
When the verifier misses a real defect, the team tries to convert the miss into a durable rule. One method extracts patterns from human review comments and pull request reverts. A candidate rule must appear in at least two pull requests and be detectable from a diff. Three agents then try to disprove it before a human reviews and approves the rule through a pull request.
The second method replays three bug-finding agents against a pull request connected to a known incident. If at least two find the bug, it is considered detectable. If all three miss a bug in a small diff, that gap is a candidate for a new rule in REVIEW.md.
Flaky tests receive a similarly strict treatment. Claude is not allowed to skip them, add sleeps, or retry until they happen to pass. If the agent cannot find the root cause, it removes the test and records why. This prevents a misleading red signal from becoming permanent background noise.
Rules must also be pruned. Test suites, verification steps, and Markdown instruction files grow unless someone is responsible for shrinking them. The team periodically removes checks and rules that no longer justify their cost.
A practical adoption sequence
The article does not prescribe a single migration plan, but its operating model suggests a sensible sequence for a team adopting agentic coding.
- First, make the product runnable by the agent. Write a short verification skill that boots the system, exercises the main flow, and returns concrete evidence. Start with one high-value user journey.
- Second, make quality visible. Capture pass or fail, the command that ran, relevant logs, a screenshot for UI work, and a clear statement of what remains unverified.
- Third, add properties and differential checks where the domain has stable invariants. Use generated inputs for parsers, serializers, state machines, migrations, and rewrites.
- Fourth, isolate the gates. Put verification rules, test definitions, and reviewer instructions on protected branches. Bind approvals to commits and make stale approvals expire.
- Fifth, roll out in observe mode. Measure false blocks before enforcement. Keep a documented escape path for legitimate exceptions.
- Sixth, evaluate the verifier against real incidents. Use known bugs, reverts, and clean controls. Measure raw, verified, and final recall separately.
- Seventh, maintain the system. Convert repeated misses into rules, route repeated rules into code or hooks when possible, and delete checks that no longer earn their runtime.
The ordering matters. Adding a strict gate before the team can produce reliable evidence mostly creates friction. Adding more tests without measuring whether the tests detect the incidents that matter mostly creates an expensive illusion of safety.
Where the argument needs more evidence
The post is unusually concrete for a first-party engineering article, but several claims remain vendor-reported. The figure of roughly 90% Claude-written code, the 10-point grader improvement, the 39-item reliability checklist, and the internal performance of the commit gate are not provided with public datasets or independent reproduction. That does not make them false. It does mean teams should translate the principles, not assume the same effect sizes.
There are also technical limits to each test category. End-to-end tests can be slow and flaky, especially across browsers, simulators, network conditions, and asynchronous UI state. Property tests are only as good as the property; a weak or incorrectly stated invariant can generate a large amount of useless evidence. Differential tests inherit the blind spots of the old system and become difficult when intentional behavior changes are not reviewed carefully.
Most importantly, a verifier built by the same model family can share the producer's assumptions. The team reduces that risk through fresh context, independent agents, human-owned checks, protected test files, incident replay, sampling, and explicit controls. None of these mechanisms is sufficient alone. Their combination is what makes the evidence more trustworthy than an agent's self-assessment.
The broader lesson is that agentic coding shifts engineering effort from typing code toward designing feedback. The scarce resource is no longer the first draft. It is the ability to establish, quickly and honestly, whether the draft works. Teams that build that layer can delegate more work, review smaller packets of uncertainty, and reserve human attention for decisions that evidence cannot settle.
Conclusion
The Claude Code team's approach can be summarized in four rules. Run the real change. Give failures a path back to the agent and, when necessary, to a person. Keep the judging system outside the change's control. Evaluate the verifier against the incidents that actually mattered.
Those rules explain why verification is more than testing. It is the infrastructure that turns parallel agent output into reviewable, reversible, and continuously improving work. Without it, more agents only create more claims to inspect. With it, more agents create more evidence that can be checked.
Source: How the Claude Code team thinks about verification, Addy Osmani, claude.dev, November 16, 2026.
Source:Anthropic (claude.dev Blog)https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/
