Workbench for Real Repos

In plain words

A demo that works once proves nothing, like a project that runs fine for your guide and crashes at the fest. So take one real task, run it twice on the same repo, once with just a prompt and once through your full agent setup. Measure the same five things both times. The table you get is the proof, not your explanation.

How it flows

A tiny example

task = "add password validation + a test"
rows = {}
for name, pipeline in [("prompt-only", plain), ("workbench", guided)]:
    result = pipeline(task)
    rows[name] = {
        "tests_actually_run": result.test_command_ran,
        "acceptance_met": result.acceptance_passed,
        "files_outside_scope": count_outside(result.touched),
    }
print_table(rows)

Notice that only the pipeline changes between the two rows, so any difference in the numbers can be blamed on the pipeline alone.

What you will learn

• How to prove your agent setup actually helps, using numbers instead of opinions.
• How to run the same task twice, one plain way and one guarded way, and compare.
• The five things worth measuring on every agent run.
• How to answer the classic pushback: "my model is smart, why all this extra setup?"

The problem, simply

Think about your final year project demo. The toy version works beautifully for your guide, everyone claps. Then the college fest comes, fifty people open it at once, and it falls apart. The demo proved nothing.

Agents have the same problem. In Module 6 and Module 7 you built a whole workbench around the agent: a scope contract, a state file, a feedback runner, a verification gate, a reviewer, a handoff packet. All of that is extra work. Somebody on your team, or your interviewer, will ask the fair question: does it actually help, or is it just ceremony?

You cannot answer that with a speech. You answer by running one realistic task through two pipelines on the same code, measuring the same five things both times, and putting the result on one page.

IMP

Note: A "pipeline" here just means the fixed order of steps the agent follows. Nothing fancy.

The idea

Same task, two roads

You pick one small but real-feeling task. Something like: add password validation to a signup endpoint, reject passwords shorter than 8 characters, return a proper error, and add a test that proves it.

Then you run it two ways.

Road one is prompt-only. Read the README, read the file, edit, say "done".

Road two is workbench-guided. Read the scope contract, read the state file, edit only the allowed files, run the acceptance command through the feedback runner, run the verification gate, run the reviewer, write the handoff packet.

The five outcomes

These are the five things you record for both runs. Nothing more, because five fits on one screen.

• tests_actually_run — did a test command really execute, or did the agent just claim it did? Most "tests passed" claims are unverifiable.
• acceptance_met — did the specific test that proves the goal actually pass? Not any test. That test.
• files_outside_scope — how many files got touched that were never meant to be touched? Scope creep is the quiet killer.
• handoff_quality — if a fresh session picks this up tomorrow, does it get a usable summary or does it start from zero?
• reviewer_total — the reviewer's overall score, a bit of judgement on top of the machine gate.

IMPRemember: measure the same five outcomes on both runs, or the comparison means nothing.

A worked example

Suppose Priya runs this on her hostel project repo, a small signup service.

Prompt-only run: the agent edits app.py, and also "helpfully" tidies README.md and scripts/release.sh. It says the tests pass. Nobody ran them. So: tests_actually_run = no, acceptance_met = no, files_outside_scope = 2, handoff_quality = poor, reviewer_total = 2 out of 10.

Workbench-guided run: the scope contract opens only app.py and test_app.py. The agent edits those two. The feedback runner really executes the test command and captures the output. The gate confirms the new test passed. The handoff packet lists what changed. So: tests_actually_run = yes, acceptance_met = yes, files_outside_scope = 0, handoff_quality = good, reviewer_total = 9 out of 10.

Now Priya is not arguing. She is showing a table.

Why this is not just theory

One team took the same coding model and, by changing only the surrounding harness, moved it from outside the top 30 to rank five on a terminal-task benchmark. Same model, different surfaces, twenty-five ranks.

Another team deleted about 80% of their agent's tools and the success rate went from 80% to 100%. Fewer ways to go wrong.

A 2026 preprint on harness engineering reports that roughly 88% of enterprise agent projects never reach production, and the failures cluster in the runtime, not in the reasoning: stale state, brittle retries, context that grows until the agent forgets the goal.

In long-context conditions, web-agent baselines that succeed 40–50% of the time can drop below 10%, mostly from infinite loops and losing the goal. The state file and the handoff packet exist to absorb that.

Be honest about where it does not help

Some tasks are genuinely faster prompt-only. A one-line lint fix. Running a formatter. A single factual question. If you hide these, your report looks like marketing and a good reviewer will smell it.

Warning

Warning: If your before/after report has zero cases where prompt-only won, nobody senior will trust the rest of it.

Build it

This script runs both pipelines against a small fake repo, measures the five outcomes, and prints a comparison. The "agent" is scripted, not a real model, so the numbers are reproducible.

# Compare a prompt-only run against a workbench-guided run on the same task.
# No network, no real model. Standard library only.

ALLOWED = {"app.py", "test_app.py"}          # the scope contract
TASK = "reject passwords shorter than 8 chars, add a test"

def run_prompt_only():
    """Agent edits whatever it feels like and claims success."""
    touched = ["app.py", "README.md", "scripts/release.sh"]
    return {
        "touched": touched,
        "test_command_ran": False,     # it only *said* tests passed
        "acceptance_test_passed": False,
        "handoff_notes": "",
    }

def run_workbench():
    """Agent reads scope, edits only allowed files, really runs the tests."""
    touched = [f for f in ["app.py", "test_app.py", "README.md"] if f in ALLOWED]
    return {
        "touched": touched,
        "test_command_ran": True,
        "acceptance_test_passed": True,
        "handoff_notes": "Added length check in app.py; test_short_password added.",
    }

def measure(result):
    """Turn one run into the five outcomes."""
    outside = [f for f in result["touched"] if f not in ALLOWED]
    return {
        "tests_actually_run": result["test_command_ran"],
        "acceptance_met": result["acceptance_test_passed"],
        "files_outside_scope": len(outside),
        "handoff_quality": "good" if len(result["handoff_notes"]) > 20 else "poor",
    }

def score(m):
    """A rough reviewer score out of 10 from the four hard outcomes."""
    points = 0
    points += 3 if m["tests_actually_run"] else 0
    points += 3 if m["acceptance_met"] else 0
    points += 2 if m["files_outside_scope"] == 0 else 0
    points += 2 if m["handoff_quality"] == "good" else 0
    return points

runs = {"prompt-only": run_prompt_only(), "workbench": run_workbench()}
print("Task:", TASK)
print()
print(f"{'outcome':<22}{'prompt-only':<16}{'workbench':<16}")
rows = {name: measure(r) for name, r in runs.items()}
for key in ["tests_actually_run", "acceptance_met",
            "files_outside_scope", "handoff_quality"]:
    print(f"{key:<22}{str(rows['prompt-only'][key]):<16}{str(rows['workbench'][key]):<16}")
print(f"{'reviewer_total':<22}"
      f"{str(score(rows['prompt-only'])) + '/10':<16}"
      f"{str(score(rows['workbench'])) + '/10':<16}")

Look at the files_outside_scope row first. The prompt-only run touched two files nobody asked for, and the agent's confident summary would never have told you. Then look at reviewer_total: the gap comes from four small checks, not from a smarter model.

Where you will see this

• Claude Code and Cursor keep project rules and a scope of allowed files, then run your test command for real before claiming a task is done.
• Teams reviewing AI-written pull requests run an automated review pass on every PR, at very large volumes, and keep the score with the diff.
• Customer-support bots at Swiggy or Flipkart scale are measured the same way: did the ticket actually get resolved, not did the bot sound confident.
• GitHub Copilot style tools ship "did the generated tests run and pass" as a first-class signal, not an afterthought.
• Any internal platform team that wants budget for agent tooling has to produce exactly this kind of before/after table.

Common mistakes

• Using a toy task. "Write a function that adds two numbers" proves nothing. Pick a task with a real file layout, a real test command, and a few files that must not be touched.
• Changing two things at once. If you change the task and the pipeline together, the comparison is meaningless. Same repo, same task, only the pipeline changes.
• Trusting the agent's own summary. "I ran the tests and they pass" is a sentence, not evidence. Capture the actual command output.
• Hiding the false negatives. Skipping the cases where prompt-only was faster makes your whole report look dishonest.
• Measuring twenty things. Five outcomes fit on one screen and get read. Twenty get skimmed and ignored.

If they ask in an interview

Q: How would you prove that your agent framework actually improves results?

A: Run the same realistic task on the same repo twice, once prompt-only and once through the full pipeline, and record five fixed outcomes: did tests really run, did the acceptance test pass, how many out-of-scope files were touched, handoff quality, and a reviewer score. Only the pipeline changes, so the difference is attributable. Then present it as one before/after table.

Q: If models keep getting better, is all this harness work wasted?

A: Models do absorb some of it over time, so specific tricks age. But today most failures come from the runtime, not the reasoning: stale state, retries, context growing until the goal is lost. Changing only the harness has publicly moved the same model up twenty-five ranks on a benchmark, so right now the leverage is in the surfaces.

Q: When would you not use the workbench?

A: For single-step things the model already knows cold: a one-line lint fix, a formatter run, a factual lookup. There the setup cost is real and the benefit near zero. Name those cases in the report, because being honest about them is what makes the rest of the numbers credible.

Try these

• Take any small Flask or FastAPI style project you already have and write down the scope contract: exactly which files an agent may touch for one task.
• Run the script above, then change run_workbench so it also touches README.md. Watch files_outside_scope and reviewer_total both move, and check the scoring feels fair.
• Add a sixth outcome, time to first meaningful edit. Decide how you would measure it cleanly without punishing an agent that reads carefully first.
• Write a one-page version of the report aimed at someone non-technical, like a placement coordinator. See what survives the cut.

Words, simply

Quick recap

• Run one real task twice, changing only the pipeline, and measure five fixed outcomes.
• The out-of-scope file count and the "did tests really run" flag catch the failures a confident summary hides.
• Report the cases where the plain prompt won too, otherwise nobody believes the cases where it lost.