Eval-Driven Development

In plain words

Solving two practice problems does not mean you will clear the placement test. Same with agents: one working demo proves nothing. So you build a question bank for your agent, run it on every code change, and block the merge if the score drops. You also put a judge inside the loop, so weak answers get refined before any user sees them.

How it flows

A tiny example

for case in cases:
    answer = agent(case.ask)
    ok, why = judge(answer, case.must)
    if not ok:
        answer = agent(case.ask, feedback=why)
    score += ok

print("merge" if score/len(cases) >= baseline - 0.05 else "blocked")

Notice the judge's feedback goes back into the agent for a second try, and one pass rate compared to a baseline decides whether the code ships.

What you will learn

• Why a demo that works once tells you nothing about production.
• The three layers of testing an agent: standard benchmarks, your own offline tests, and live checks.
• The propose-judge-refine loop that fixes an answer before the user sees it.
• How to run agent tests automatically on every code change, like a build gate.

The problem, simply

Think about how you prepare for a placement drive. You solve two problems on a practice site, both work, and you feel ready. Then the actual TCS or Amazon online test comes with a tricky edge case, and your code fails. Nothing was wrong with your practice — it was just too small.

Agents have the same disease. You build one, show it to a friend, it summarises the PDF correctly, everyone claps. Then real users arrive with weird inputs and slow tools, and the agent quietly breaks.

See, the problem is that one demo is one test case. Production is thousands of test cases you have never seen. And unlike normal code, an agent can give a slightly different answer every time you run it, so "it worked when I checked" is not proof of anything.

So the fix is not more clever prompting. The fix is building a proper question bank for your agent, and running it every single time you change something. That practice is called eval-driven development. "Eval" is just short for evaluation — a test for an agent.

The idea

Three layers of evals

You need three different kinds of tests, because each one answers a different question.

Layer 1: static benchmarks. Ready-made public test sets. SWE-bench is a set of real GitHub bugs an agent must fix. GAIA tests general problem solving. WebArena and OSWorld test agents that click around websites and desktops. BFCL tests tool calling. These answer "is this model broadly capable?" — useful when choosing between models.

Warning

Warning: Public benchmarks leak. Models sometimes saw the answers during training, so scores look better than the real ability. An audit of SWE-bench found around 32% of cases had solution leakage. Always use the cleaned, human-verified versions and say which version you used.

Layer 2: custom offline evals. These are your own test cases, on your own product's shape. Three common styles:

• Execution-based — actually run what the agent produced and check it. Run the patch, run the tests.
• LLM-as-judge — a second model reads the output and scores it against a rubric.
• Trajectory-based — compare the agent's sequence of steps against a known good sequence. Studies of desktop agents found the top agents take roughly one and a half to three times more steps than a human expert would.

Layer 3: online evals. These run on real traffic: session replays so you can watch what happened, alerts when a guardrail trips, and per-step cost and latency tracking so you notice the day your agent starts burning ₹12 per query instead of ₹2.

The propose-judge-refine loop

Now here is the trick that makes evals do more than just report failure. Instead of only grading at the end, you put a judge inside the agent itself.

One part generates the answer. A second part judges it against your rules. If it fails, the first part gets the judge's feedback and tries again, up to some limit. This is called evaluator-optimizer — the general form of self-refinement, where a model critiques and improves its own work.

Suppose Priya builds a support agent for a college fee portal. A student asks for a refund and the agent drafts a reply. The judge checks three things: is the refund rule quoted correctly, is the tone polite, does the reply stay in scope. Round one fails — the agent invented a 7-day refund window that does not exist. Round two quotes the real 15-day rule and passes. The student never sees round one.

IMP

Important: A judge model can hallucinate just like the agent. Give the judge real tools — let it look up the actual rule document or run the actual test — instead of asking it to judge from memory.

Where the evals should live

The 2026 practice is simple and boring, which is why it works.

Keep eval cases in the same repository as your agent code. Run them in CI on every pull request. Store a baseline score from the last known-good version. Block the merge if the score drops more than your allowed threshold, say 5%.

IMPRemember: every time something breaks in production, your first job is not to fix it — it is to write the eval case that reproduces it. Then fix it. That way the same bug can never come back silently.

Also map each safety guardrail to at least one eval case. If you cannot point at a test for a rule, that rule is decoration.

Build it

"""A tiny eval harness: cases, a fake agent, propose-judge-refine, and a CI gate."""

# Each case: input, the words the answer must contain, and which layer it belongs to.
CASES = [
    {"id": "refund_rule", "ask": "refund window", "must": ["15 days"], "layer": "custom"},
    {"id": "fee_amount",  "ask": "semester fee",  "must": ["12000"],  "layer": "custom"},
    {"id": "hostel_rule", "ask": "hostel timing", "must": ["10 pm"],  "layer": "custom"},
    {"id": "tool_use",    "ask": "unknown tool",  "must": ["cannot"], "layer": "benchmark"},
]

FACTS = {"refund window": "15 days", "semester fee": "12000",
         "hostel timing": "10 pm", "unknown tool": "cannot do that"}

def fake_model(ask, feedback):
    """Stand-in for a real model. Round 1 guesses; feedback makes it look up the fact."""
    if feedback is None:
        return "I think it is about 7 days, roughly."
    return "As per the rules, it is " + FACTS.get(ask, "not listed") + "."

def judge(answer, must):
    """Grounded judge: checks the answer against known facts, not against a vibe."""
    missing = [word for word in must if word.lower() not in answer.lower()]
    return (not missing), ("missing: " + ", ".join(missing) if missing else "ok")

def run_case(case, max_rounds=3):
    feedback = None
    for round_no in range(1, max_rounds + 1):
        answer = fake_model(case["ask"], feedback)
        passed, why = judge(answer, case["must"])
        if passed:
            return True, round_no
        feedback = why          # the judge's note goes back to the proposer
    return False, max_rounds

BASELINE = 1.0                  # pass rate of the last known-good version
ALLOWED_DROP = 0.05             # fail the build on more than a 5% regression

passed_count = 0
for case in CASES:
    ok, rounds = run_case(case)
    passed_count += ok
    print(f"{case['id']:<12} {case['layer']:<10} "
          f"{'PASS' if ok else 'FAIL'} in {rounds} round(s)")

score = passed_count / len(CASES)
print(f"\npass rate {score:.2f}  baseline {BASELINE:.2f}")
print("CI GATE: MERGE" if score >= BASELINE - ALLOWED_DROP else "CI GATE: BLOCKED")

Look at the round count first. Every case needs two rounds: round one is a guess, and the judge's note is what pulls the model to the real fact. Then look at the last two lines — that is the whole CI gate. One number compared with a stored baseline decides whether the code may merge.

Where you will see this

• Coding agents like Claude Code and Cursor are measured on fixed sets of real repository bugs before a new version ships.
• GitHub Copilot suggestions are checked by running the project's own test suite, which is execution-based evaluation.
• Customer-support bots at banks and telecom companies replay yesterday's real chats against a new prompt before rolling it out.
• Swiggy or Zomato style assistants track per-query cost and latency in production, which is an online eval running all day.
• ChatGPT-style products use graded rubrics over sampled conversations to catch tone and safety drift.

Common mistakes

• No baseline stored. A pass rate of 84% means nothing alone. Without last week's number you cannot tell improvement from decay.
• Judge with no grounding. If the judge only reasons from memory, it invents facts as confidently as the agent does, and you get a green build on a wrong answer.
• Over-fitting to the eval set. If you tune the agent until those 50 cases pass, you have optimised for 50 cases, not for users. Rotate in fresh cases regularly.
• Flaky cases. Non-deterministic tests fail randomly, everyone starts ignoring red builds, and a real regression walks straight through. Pin random seeds and snapshot the tool state.
• Evals in a separate place. If the tests live in some dashboard nobody opens, they run once a quarter. Keep them beside the code so they run on every change.

If they ask in an interview

Q: How would you test an AI agent, given that it does not give the same answer every time?

A: I would not test for an exact string. I would test for properties — does the answer contain the right fact, does it call the right tool, does it stay within a step budget. And I would run each case a few times with a pinned seed, then track the pass rate against a stored baseline instead of judging a single run.

Q: What is the evaluator-optimizer pattern?

A: One component proposes an answer, a second component judges it against a rubric, and if it fails the feedback goes back to the proposer for another attempt, up to a fixed number of rounds. It is self-refinement made into a reusable wrapper, so you catch bad output before the user does rather than after.

Q: Public benchmark scores look great but the agent fails for our users. Why?

A: Benchmarks answer whether a model is broadly capable, not whether it does my product's job. There is also contamination, where benchmark solutions leaked into training data and inflate scores. So I use benchmarks only to compare models, and rely on my own offline and online evals for shipping decisions.

Try these

1. Take the harness above and add three cases of your own that currently fail. Watch the CI gate turn from MERGE to BLOCKED, then make them pass.
2. Write a judging rubric for a domain you know, with three dimensions such as factual, tone, and scope. Score ten sample answers by hand and note where two people would disagree.
3. Add a step-efficiency check: record how many rounds each case took and fail any case that needs more rounds than a limit you choose.
4. Break one case deliberately by making the fake model return a random answer, run the harness five times, and see how a flaky test destroys your ability to read the score.

Words, simply

Quick recap

• Demos prove nothing; a running eval suite at three layers is the only real proof.
• Put a grounded judge inside the loop so bad answers get fixed before the user sees them.
• Keep evals next to the code, run them on every change, and block merges on regression against a stored baseline.