What Is Eval-Driven Development? How to Improve AI Agents

Most agent development starts the same way: write a prompt, try a handful of questions by hand, adjust the prompt, and repeat. That loop works while failures are obvious. It becomes much less reliable after the agent grows into a multi-step system, because a change that repairs one case can quietly damage another. At that point, “the latest answer looked better” is not enough evidence to support a release decision.

Eval-driven development (EDD) replaces that intuition-heavy loop with a stable measurement system. The team defines what good behavior means, writes an evaluation before implementing the capability, records a baseline, and compares every subsequent experiment against it. LangChain describes the process in three steps:

1. Define what good looks like for the agent.
2. Write the evaluation before building the feature.
3. Make the change, then compare experiments.

The idea borrows from test-driven development, but it cannot be copied mechanically. Agent behavior is stochastic, language admits multiple correct answers, and the agent chooses its own execution path. EDD therefore evaluates a distribution of runs over a dataset rather than asserting that one exact string equals another.

Key takeaways

• EDD is controlled hill climbing. Build the agent while measuring it against a consistent set of metrics, and use evidence to decide whether the next change is progress.
• Start with a short behavioral spec. Say when the behavior should occur, what the agent must do, what it must not do, and which existing behavior must remain intact.
• Compare experiments, not isolated responses. A higher average can hide a serious regression in one important example.
• Evaluation is a continuous production system. Run fast checks in CI, sample and score real traffic after launch, then turn newly discovered failures into dataset examples.

What is eval-driven development?

In test-driven development, a developer writes a failing test first and then writes code until the test passes. EDD preserves the order of “define success first,” but changes the unit of verification from a deterministic test case to a scored set of examples.

Traditional software tests are a poor fit for agents for three reasons:

• Outputs vary. The same question can receive a different response on each run.
• More than one answer can be right. Two natural-language responses may express the same correct result with different wording.
• The agent chooses its own steps. It may call an unnecessary tool, skip a useful one, and still reach a plausible final answer.

An evaluation handles this uncertainty by scoring many runs across a dataset. The evaluator can inspect the final response, a tool call, an intermediate step, the complete sequence of actions, or the state left behind in an environment. Deterministic properties can be checked with code; subjective properties can be graded with an LLM judge. In EDD, those scores become the operational definition of success, and the team builds toward them.

Dimension	Test-driven development	Eval-driven development
Primary artifact	A deterministic test	A dataset plus one or more scored evaluators
Expected result	Usually one exact assertion	A pass rate, score, or behavioral property across examples
Failure diagnosis	Stack trace and test output	Trace of model calls, tool calls, state changes, and scores
Core risk	A broken contract	Regression, overfitting, noisy judges, and incomplete coverage

Step 1: Define what good looks like for the agent

The first task is not to write a better prompt. It is to define what the agent should do and list the behaviors required to get there. Those behaviors might include answering from a knowledge base, taking an action on a user's behalf, or recognizing when to ask a follow-up question. The requirements for each behavior together form a product requirements document for the agent.

EDD works on one behavior at a time. A narrow target makes it possible to hill climb without several simultaneous changes obscuring which one caused an improvement or regression. The behavior usually comes from a feature about to be built or a failure found in historical runs.

Before writing code, write a short spec with four parts:

• When: the situation in which the behavior should occur.
• Do: the required action.
• Don't: an adjacent behavior that must be avoided.
• Keep intact: existing behavior that must not regress.

The wording must be precise enough to score as pass or fail. “Be helpful” is not a behavioral requirement. A tool-routing decision, a required state transition, or an explicit refusal can be evaluated.

A concrete internal-docs example

Suppose an internal documentation agent answers employee questions with two existing tools: search_docs for company documentation and web_search for public information. The team plans to add a create_ticket tool so the agent can file IT requests.

The behavior spec can be written as follows:

• When: a user reports something broken, such as “My laptop won't connect to the VPN.”
• Do: call create_ticket.
• Don't: file a ticket for a how-to question such as “How do I connect to the VPN?”
• Keep intact: search_docs and web_search still handle the requests they handle today.

The evaluation is defined against these guidelines even though the new tool does not exist yet. That deliberate gap is what creates a failing target for implementation.

Step 2: Write the evaluation before building

Build a dataset

A dataset is a collection of examples. Each example contains an input and, optionally, a reference or “golden” output. Real agent runs are one of the best input sources: users phrase requests in unexpected ways and introduce edge cases that developers would not invent on their own. The closer the dataset is to real traffic, the more likely it is that a good evaluation score will transfer to production.

If there are no production runs yet, start from scratch. For the docs agent, a small set is enough: ticket requests, questions that should still use search_docs, and questions that should still use web_search. Including the existing routes from the beginning makes it possible to detect collateral damage when the new tool is introduced.

Datasets can also be expanded deliberately. If there are few examples from frustrated users, existing cases can be rewritten into angrier messages to test whether tool routing remains stable. Synthetic variation is useful when it increases behavioral coverage, not when it merely inflates the number of rows.

Write an evaluator

Turn the definition of “good” from the spec into an evaluator. Choose the scoring method according to the property:

• Use a code evaluator when the check is deterministic, such as whether a required tool was called or whether an environment ended in a valid state.
• Use an LLM-as-a-judge evaluator when the property is semantic or open-ended, such as whether an explanation is faithful to the retrieved evidence.

Also decide what part of the run to inspect. For the docs agent, final prose is not the right target. The important behavior is tool choice, and that can be checked in code: an example passes when the tool in the run matches the tool in the reference output.

Evaluate the environment, not just the transcript

More complex agents need a broader evaluation surface. For an agent that changes an environment, such as a coding agent in a sandbox, a few inspected tool calls and a plausible final response are insufficient. The evaluator must inspect the state the agent leaves behind, and every run needs a clean environment so one trial does not contaminate the next.

Harbor is an open-source framework for running agents on environment-changing tasks. Each task runs in an isolated sandbox and includes a test script that grades the resulting state. It is also the harness behind Terminal-Bench. Benchmarking with a system such as this follows the same EDD loop: define tasks and tests first, run a baseline, and compare experiments while changing the agent.

This environment-state view is especially important for robotics and embodied agents. A robot can produce a fluent answer and still fail the task if an object was never grasped, a collision changed the scene, or an unsafe intermediate action occurred. The final screenshot or text response is only one observation; the evaluator also needs the resulting physical or simulated state, relevant safety constraints, and enough instrumentation to reconstruct the trajectory.

Run a baseline

Run the current agent on every example and save the results as an experiment. An experiment is one pass of an agent over a dataset, with each example's input, output, tool calls, state, and evaluator scores recorded together. Keeping experiments in one system makes versions directly comparable.

The baseline must be recorded before the change. For the docs agent, ticket examples fail because create_ticket does not exist yet. The other examples establish the behavior that already works, so the next experiment can show whether adding the new tool damages old routes.

Step 3: Make the change and compare

Now implement create_ticket and run the same evaluation again. EDD is a hill-climbing loop: evaluate, compare, adjust, and repeat. Every change produces a new experiment that is compared with the previous one using objective evidence.

The first comparison may look encouraging. Ticket requests now pass, so the average score rises. But an important how-to question has regressed: “How do I connect to the VPN?” now files a ticket instead of searching the docs.

This is why an average alone is not a release gate. The new behavior improved several examples while overwriting one previously correct behavior. Inspecting only the aggregate would have hidden the regression.

Use traces to find the cause

A trace records a single run, including every model call, tool call, input, output, and relevant execution detail. Opening the failing trace is usually faster than guessing from the final response.

In the example, the trace shows that the tool description for create_ticket says to use it for “VPN and laptop issues.” That language is too broad. Any VPN question matches, including questions that documentation can answer without creating a ticket.

The fix is to narrow the tool description to requests that require IT to act. Run the evaluation again. This time all eight examples pass.

Do not optimize only for the examples in front of you

A perfect score on a small dataset is a useful signal, but 100% is not automatically the right goal. The tool description was tuned while looking at those exact examples. If the team keeps changing the agent until every visible case passes, it may overfit the dataset rather than the behavior. The eval can look perfect while requests with unfamiliar phrasing still fail.

Keep a held-out set that is not inspected during iteration. If scores rise on the tuning set while remaining flat on the held-out set, the team is likely overfitting. Use the held-out set as an estimate of generalization, not as another source of examples to tune against.

Multi-step agents can fail at any point, so evaluation should not stop at the final answer. Useful evaluation units include an individual step, the complete sequence of tool calls, and a multi-turn conversation grouped into a thread.

Keep evaluation running after release

Put fast checks in CI

As deployment accelerates, manual evaluation is easy to skip and old examples are easy to forget. Automated evaluation prevents untested changes from reaching users and catches regressions before release.

The right suite depends on the system. Fast deterministic checks can block a bad pull request. Slower LLM-judge evaluations can run nightly or before a release. Dataset splits let developers run only the examples for the current behavior while iterating, then switch to the full suite before shipping.

Evaluation frameworks can run next to ordinary tests through integrations such as pytest, Vitest, or Jest. Every run should still be recorded as an experiment so that CI results, manual experiments, and production signals remain comparable.

Evaluate production traffic

A pre-deployment dataset covers only the cases the team thought to collect. Production exposes the rest. A complete operating loop has several complementary layers:

• Online evaluators score live traces. Code checks and LLM judges can run on a sampled share of traffic to control cost.
• Annotation queues route ambiguous or flagged runs to humans. They also provide the labels needed to verify that an LLM judge agrees with people before it is trusted.
• Dashboards track quality scores, user feedback, cost, and latency over time, making release-level regressions visible.
• Trace clustering groups large numbers of runs into recurring failure patterns that would be invisible in individual examples.
• Failure-to-dataset loops turn a reviewed production failure into a new example with an assertion, so the same bug becomes a permanent regression test.

For embodied systems, the production signal should connect the digital trace to the physical outcome. A robot fleet may log planner decisions, model calls, sensor summaries, safety interventions, task success, recovery behavior, and human takeover. Sampling only successful episodes or polished text responses creates a dangerously incomplete eval set.

Five common mistakes

1. Defining success too loosely. “The response is helpful” cannot be scored consistently. Define a behavior that can be judged as pass or fail.
2. Writing the eval after the fix. Once the new output is visible, it is too easy to write an evaluator that passes by construction. Write it first and confirm that it fails on the current version.
3. Checking only the average. A higher average can hide a regression on a critical example. Review per-example results before accepting the change.
4. Adding evaluations that do not matter. Every eval pushes the system in some direction. Tie each one to a concrete failure or feature instead of maximizing metric count.
5. Trusting an unchecked judge. Compare an LLM judge against human labels first. Ask narrow questions and prefer pass/fail decisions over an ambiguous 1-to-10 score.

An operational EDD checklist

Stage	Question the team must answer	Evidence produced
Specify	What exact behavior are we changing, and what must not regress?	A four-part behavioral spec and dataset examples
Measure	Can this property be checked in code, or does it require a calibrated judge?	A deterministic evaluator or a human-validated judge
Baseline	What does the current version score before the change?	A recorded experiment over the agreed dataset
Compare	What improved, what regressed, and why?	Per-example comparison and relevant traces
Ship	Does the change generalize and pass the full release gate?	Held-out results, CI evidence, and production monitoring

EDD does not remove judgment from agent development. It changes where judgment is applied. The team still decides what good behavior means, which tradeoffs matter, and which examples are important. Those decisions become explicit in a spec, a dataset, and an evaluator instead of remaining implicit in whoever happens to test the latest prompt.

For a single demo, hand testing may be sufficient. For an agent that will keep changing, serve users, and make decisions in an environment, the evaluation system is part of the product. A reliable agent is not one with a convincing latest response; it is one that can repeatedly demonstrate improvement without silently erasing behavior that already worked.

Source: LangChain Blog, “What Is Eval-Driven Development? How to Improve AI Agents”, by Divya Santhanam, published November 9, 2026.