Catching Bugs Before They Land: How Meta Scaled JiTTests Around Human-on-the-Loop Review
Meta turned Catching JiTTests into production infrastructure: two test-generation paths, layered false-positive triage, and human-on-the-loop intent review.
Every code change can improve a platform while also making it more complex. Across hundreds of millions of lines of continuously evolving code, traditional testing struggles to answer a more immediate question: what observable behavior did this diff change, and was that change intended? Meta's Catching JiTTests moves quality assurance from “ask a model to inspect the code” to “ask a model to produce a test that can actually fail.” The approach has now moved from research prototype into production infrastructure.
The important idea is not simply that a language model writes tests. The system first treats a diff as potentially faulty and constructs tests that distinguish the parent version from the changed version. A test becomes a “weak catch” only when it passes on the parent and fails on both a mutant and the real diff. Rule-based checks, an ensemble of LLM judges, and engineers then decide whether the observed behavioral difference is a genuine bug.
This deep dive follows Meta's engineering post and the FSE 2026 paper, Just-in-Time Catching Test Generation at Meta, covering the architecture, production results, organizational design, and the limits that the headline numbers do not settle.
1. Why another review pass is not enough
Most AI code review still follows the shape of human review: a model reads a diff, uses repository context and prior experience, and leaves a comment saying that something might be wrong. It may be much faster than waiting for a colleague, but the form of evidence has not changed. A person still has to decide whether the suggestion is credible.
Catching JiTTests takes a different route. It produces an executable test, not only a semantic guess. A useful test first shows that the parent behavior is normal, then shows that the changed code behaves differently. The review signal moves from “the model sees a possible risk here” to “this input produces a different result,” turning the issue into a behavior that can be run, reproduced, and discussed.
This does not make test generation cheaper than code review. It requires more computation, stronger execution infrastructure, and additional false-positive filtering. Meta's bet is that if the extra work is concentrated on high-risk diffs and reduces the time engineers spend investigating noise, the tradeoff is worthwhile.
| Dimension | LLM-based code review | Catching JiTTests |
|---|---|---|
| Primary output | A comment that “this might be a bug” | A failing test and an observable behavioral difference |
| Coverage mechanism | Issues inferred from code semantics | Behavior exercised through execution |
| Human decision | The author judges whether the suggestion is correct | The author confirms whether the observed behavior is intended |
2. Weak catches and strong catches
JiTTests are generated just in time and are designed to fail, exposing bugs before code lands. But a failing test is not automatically a real bug. Meta separates the candidates into two levels:
- Weak catch: a test fails on the real diff and may reveal a bug, but it may also be a false positive.
- Strong catch: later evidence and human confirmation establish that the behavioral change is genuinely a bug, making it a true positive.
The hard part is not producing more failing tests. It is extracting a small set of strong catches from a large pool of weak catches while keeping engineers out of a flood of noise. A system that creates a thousand failures for humans to inspect has only replaced a compute bottleneck with a human one.
3. Two complementary generation workflows
Meta runs two workflows that do not require the system to know in advance exactly how a change should be written. They generate distinguishing tests from two different angles.
3.1 The “dodgy diff” workflow
This path treats every diff as a mutant, effectively assuming that it contains a bug. It uses mutation-guided, LLM-based test generation to create tests that distinguish the parent code from the diff. It does not try to infer intent; it tries to maximize the pool of weak catches.
The pipeline first generates extra tests that can kill a mutant, checks that those tests pass on the parent, and then runs them against both the mutant and the real diff. Only tests that pass the parent and fail the mutant and the diff become weak catches.
The parent baseline matters. A test that already fails on the parent may be exposing an old bug, an environment problem, or a broken test, so it is not valid evidence that the current diff introduced the behavior.
3.2 The intent-aware workflow
The second path asks an LLM to infer what the change intends to accomplish from the code, diff title, and summary. It then describes ways an implementation could fail to realize that intent and constructs mutants representing those risks.
For example, a change that restricts access to a range might go wrong by reversing a boundary, mishandling a default, bypassing a check on an exception path, or failing to update a caller. Tests are constructed around those risks and then run against the real diff.
The two paths encode different assumptions: one distrusts intent and searches behavior broadly; the other uses intent to narrow the search. Running both preserves coverage of unanticipated risks while improving the chance of finding bugs directly connected to the author's goal.
4. The five-stage production workflow
| Stage | What happens | Why it exists |
|---|---|---|
| 1. Target selection | A Diff Risk Score-like mechanism selects the previous day's risky diffs and runs the system overnight on spare capacity | Spend expensive testing effort where severe regressions are most likely |
| 2. Test generation | Dodgy diff and intent-aware workflows run in tandem and harvest weak catches | Combine broad behavioral coverage with intent-directed search |
| 3. LLM-as-Judge ensemble | Several state-of-the-art models review test code, failure output, the diff, and inferred intent | Use semantic understanding to filter likely false positives |
| 4. RubFake | A deterministic rule-based assessor checks known non-bug failure modes | Add verifiable, repeatable constraints that model judgment cannot provide |
| 5. Human-on-the-loop | Only candidates with sufficiently high rankings reach engineers | Reduce per-item supervision while retaining the final intent judgment |
Stage 1: decide where compute is worth spending
The system does not apply the same effort to every change. It runs overnight against diffs submitted the previous day that are considered risky by a mechanism similar to Diff Risk Score and trained on previous code changes. The point is not sophistication for its own sake: test generation, mutation, and repeated execution all have real costs. Risk routing makes the system operationally viable.
Stage 2: generate weak catches
The dodgy diff path expands the number of candidates. The intent-aware path steers search toward boundaries where the author is most likely to make a mistake. Any test that fails on the real diff but passes on the parent is recorded as a candidate behavioral difference. The system does not yet know whether it is a bug.
Stage 3: an ensemble of LLM judges
The LLM-as-Judge step does not hand final authority to one model. An ensemble examines four kinds of evidence: the weak-catch test, the failure code and trace, the parent-to-child diff, and the inferred intent.
The ensemble reduces the variance of a single semantic judgment, but Meta also acknowledges the limitation: LLM judgments are semantically rich, yet non-deterministic and difficult to verify. They cannot decide alone whether an engineer should be interrupted.
Stage 4: RubFake adds determinism
Meta complements the judges with Rule-Based False-Positive Killing Environment, or RubFake. The rule-based assessor weeds out known false-failure conditions involving the environment, dependencies, test construction, or execution path. Its conclusions are easier to reproduce and audit than a model's opinion.
The judges provide contextual interpretation; RubFake supplies a hard floor. Their combined ranking, rather than either signal alone, determines which candidates deserve attention. This is one of the most useful engineering lessons in the system: generation expands recall, deterministic checks provide a floor, and people make the product-intent decision.
Stage 5: keep the human on the loop
Meta distinguishes human-in-the-loop from human-on-the-loop. The first requires a person to review every output. The second has people define success criteria, monitor aggregate outcomes, and intervene on exceptions and high-value cases.
Engineers usually do not need to read the generated test. They only need to confirm whether the observed behavioral change is one the change intended to make. If it is not, the test may have found a real defect. If it is, the test may have the wrong expectation.
| Dimension | Human-in-the-loop | Human-on-the-loop |
|---|---|---|
| Operating model | A person reviews every action or output | A person defines success criteria; AI executes autonomously |
| Bottleneck | Human throughput determines system throughput | People set guardrails and review aggregate outcomes |
| Scaling path | Grows linearly with headcount | Grows with compute; people handle exceptions |
| Oversight | Per-item supervision | Policy guidance and exception handling |
5. What the production results show
At the time of the post, 589 Meta engineers, or 64% of those reached, had engaged with Catching JiTTests. Those engineers reported actionable findings with a 60% true-positive rate. In the first half of 2026, the system helped catch 74 serious privacy SEVs at the diff stage, before production. Meta also observed a 20% reduction in overall privacy SEV occurrence during that period.
The paper provides more detailed early measurements: the study analyzed 22,126 generated tests. Code-change-aware methods improved candidate-catch generation by roughly 4x over hardening-style tests and 20x over tests that fail only by coincidence. Rule-based and LLM-based assessors reduced human review load by 70%. Of 41 candidates initially reported to engineers, eight were confirmed as true positives; four would have led to serious failures if left uncaught.
| Metric | Result | How to read it |
|---|---|---|
| Engineer engagement | 589 people, 64% of outreach | The system entered a real workflow rather than remaining an offline experiment |
| Actionable findings | 60% true positive | Attention must still be paid to noise and the exact evaluation protocol |
| Serious privacy SEVs | 74 caught at the diff stage | Value appears before production, but “played a key role” is not sole causation |
| Privacy SEV volume | 20% reduction in the period | The post reports an association, not a controlled causal estimate |
| Candidate generation | 4x improvement over hardening tests | Derived from the paper's early set of 22,126 tests, not a universal guarantee |
| Human load | 70% reduction from assessors | False-positive control is central to scalability, not an afterthought |
The most important number is not 74 by itself. It is where the discovery happened. The tests ran before merge, while the author still had context and fixes were cheap. Once a privacy defect reaches production, the organization must absorb monitoring, incident response, remediation, and trust costs that are not equivalent to a pre-merge correction.
6. Why this matters more in the age of agents
Agents increase the speed at which teams write code, modify tests, and open pull requests. If humans remain the final per-item gate, development will simply arrive at the human bottleneck faster. Catching JiTTests does not remove people. It shifts their work from repetitive reading toward criterion setting, aggregate auditing, and intent decisions.
Three principles transfer to other engineering organizations:
- Verify behavior instead of producing more opinions. “This might be a bug” is difficult to adjudicate at scale. A test that passes on the parent and fails on the change gives the reviewer an actionable artifact.
- Treat false-positive control as a core feature. More generated tests do not automatically make a system useful. Without layered triage, generation becomes another engineer backlog.
- Route by risk rather than running the heaviest pipeline everywhere. High-risk diffs deserve more compute; low-risk changes should remain fast. The goal is to match verification intensity to potential loss.
The method also shifts what “testing” means. A conventional regression test is expected to pass when created and continue preserving behavior. A catching test is designed to fail on a suspicious change and demonstrate that the change introduced different behavior. It does not replace the regression suite; it adds a dynamic evidence layer before the change lands.
7. Five questions behind the headline numbers
1. The results have not been independently reproduced
The blog and paper report results from Meta's codebase, tooling, and engineers. The 22,126 generated tests and 64% engagement rate are substantial, but they do not establish that another organization will see the same return. Meta has a very large codebase, mature mutation infrastructure, dedicated assessors, and access to model capacity; those assets are part of the system's capability.
2. A true positive is not the same as complete recall
The reported rate describes the share of reached candidates that engineers found valid. It is not a measure of all bugs in all diffs. A diff can still contain requirement misunderstandings, cross-system semantic errors, data-distribution failures, or performance and concurrency defects that the current test environment cannot expose. A failing test is strong evidence, but only for behavior that an oracle and execution environment can observe.
3. Costs are distributed across several infrastructure layers
Generation needs model calls, mutation validation needs build and execution capacity, intent inference needs diff context, and every actionable finding passes through judges and rules. Overnight spare capacity can reduce cost, but the post does not report model tokens, execution time, storage, and engineer confirmation time per strong catch. A general ROI claim therefore remains premature.
4. Human-on-the-loop requires mature governance
Per-item review can be reduced only if the system handles most noise reliably and the organization knows when to tighten thresholds. A loose threshold consumes engineering time; a tight one misses low-scoring but real bugs. Aggregate oversight also needs monitoring across languages, code areas, and risk classes, because a single global true-positive rate can conceal local failures.
5. The privacy SEV reduction is not a controlled causal result
Meta says Catching JiTTests played a key role and observed a 20% reduction over the period. This supports the claim that the system participated in risk reduction, but it does not rule out concurrent security programs, process changes, or classification effects. The defensible conclusion is that the system has become one part of the privacy defense line and appears alongside better outcomes.
8. Six practical steps for adopting the idea
- Start with one class of high-risk changes. Security boundaries, permissions, data access, or billing logic are better first targets than every diff in the repository.
- Establish parent-versus-change execution. Without a trustworthy parent baseline, a failing test cannot prove that the current change introduced the behavior.
- Measure recall, false positives, and human time separately. Counting generated tests rewards noise; counting true positives alone hides missed bugs.
- Keep deterministic filtering. Let rules handle environment, dependency, and known false-failure modes; reserve language models for semantic judgment.
- Ask engineers to confirm intent, not read test implementation. Present the input, parent behavior, changed behavior, and failure trace, then ask whether the change was intended.
- Turn escaped bugs into evaluation cases. After every production incident, ask why no test was generated or why the candidate was filtered out.
9. Conclusion: turn the model from a commentator into an experimenter
The central contribution of Catching JiTTests is not that an LLM can write tests. It is that the model is placed inside an engineering loop that can check the model itself. The model proposes candidates, execution verifies behavioral differences, rules and judges rank them, and humans decide whether the product intent is correct. Each layer does what it is best suited to do, and none is asked to be the sole source of truth.
As agents increase the volume of code, this pattern of generate, execute, triage, and govern will become more important. The strongest AI software systems will not merely offer opinions faster than people. They will turn opinions into experiments, turn experiments into reviewable evidence, and keep people at the point where society and product teams decide what behavior should happen.
Sources and further reading
Original post: Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale, Engineering at Meta, October 29, 2026.
Paper: Just-in-Time Catching Test Generation at Meta, FSE 2026 Industry Track.