Turn Feedback into System

In plain words

Think of your hostel mess. If you complain about watery dal only in conversation, tomorrow it is watery again. If the complaint becomes a written measure and a check before serving, it is fixed forever. Corrections you give a coding agent work the same way: in chat they fix one run, but moved into a test, a rule, or a script they fix every run after that.

How it flows

A tiny example

for c in corrections:
    cause = find_root_cause(c)
    key = fingerprint(cause, c.area)
    if key in controls:
        controls[key].count += 1
    else:
        controls[key] = add_control(layer_for(cause))

print([k for k, v in controls.items() if v.count >= 2])

Notice the fingerprint: two differently worded complaints with the same cause become one control, not two rules.

What you will learn

• Why every correction you give an agent is actually free information about your setup.
• Where to put that lesson so the agent never repeats the mistake.
• How to merge repeated lessons instead of piling up rules.
• How to delete old rules that no longer protect anything.

The problem, simply

Think about your hostel mess. Every evening someone complains that the dal is too watery. The cook nods, adds a bit more dal that day, and the next day it is watery again. Nothing changed, because the complaint lived only in that conversation.

Now suppose the mess committee writes it down: 200 grams of dal per 10 people, and one student checks the pot before serving. The complaint has become part of how the mess runs. You never say it again.

Working with a coding agent is exactly this. You tell it "do not touch the README", "the output should be JSON, not a paragraph", "install the requirements first". It obeys, that run goes fine, and tomorrow it does the same thing again. Your correction fixed one run only.

See, the problem is not that the model is weak. The problem is that your correction is stored in chat, and chat is thrown away. Anything you want to survive must be moved into the project itself.

The idea

A correction is evidence, not a complaint

When you type "do not edit that file", you have discovered that your scope boundary is not written anywhere the agent can see. When you type "wrong output shape", you have discovered that no example and no test defines that shape. When setup fails again, you have discovered that your setup steps live in your head, not in a script.

So stop treating a correction as "the agent is dumb". Treat it as a small bug report about your working setup.

Tip

Tip: After every session, look back at what you had to say twice. Those repeated sentences are your to-do list.

Push the lesson to the earliest layer

Not every lesson belongs in the same place. The rule of thumb: put it as early as possible, where the mistake becomes impossible instead of merely discouraged.

A test that fails loudly beats a paragraph saying "please remember". A permission rule that blocks a folder beats a polite request. Politeness is not enforcement.

A worked example

Suppose Priya is building the payments page for a college fest site. Her agent keeps returning the amount as "₹499", a string, when the API needs 49900 paise as an integer.

She corrects it in chat. It happens again. Third time, in a different file.

Now she asks: what is the root cause? Not "the agent likes rupee symbols". It is that nowhere in her project does an example show the correct shape. So she adds one sample request and response, a test asserting the amount is an int, and one line in the project instructions pointing at the example.

Now the test fails immediately if it slips. Priya never types that correction again.

Cause is not the same as symptom

"The agent edited the README" is a symptom. The cause could be many different things: the task said "work in the repo root", or docs were assumed safe, or the plan mixed code and documentation together, or two agents were working on overlapping files.

Each of those needs a different fix. A rule that only repeats the symptom lets the next slightly different case slip right through.

Merge duplicates with a fingerprint

You will report the same problem in different words on different days. "Wrong money format", "amount came as text", "paise not used". Three sentences, one cause.

So give each control a fingerprint: a short key built from the cause and the area, not from your exact words. Before adding a new control, check if that fingerprint exists. If yes, just bump its count. Otherwise your instructions file becomes an essay nobody reads.

Rules also go stale

Old rules do real damage. They contradict newer rules and describe a system you no longer have.

So every control should carry a review date and an owner. Delete or rewrite it when the architecture changed, when a stronger test now covers it, when the failure has not come back for a long time, or when it causes more friction than the risk it prevents.

IMPRemember: the goal is not the longest instructions file. It is the smallest set of controls that still holds your hard-won judgment.

Warning

Warning: Do not promote every one-off preference into a permanent rule. Promote when it repeats, or when a single occurrence is expensive enough to justify the cost.

Build it

Here is a tiny ratchet. It takes a list of corrections, works out where each lesson belongs, merges duplicates by fingerprint, and prints what should be added to the project.

import hashlib
import json

# Which kind of root cause goes to which layer.
LAYERS = {
    "wrong_result": "test",
    "off_scope": "scope_rule",
    "setup_miss": "script",
    "bad_format": "example_and_validator",
    "convention": "project_instruction",
    "product_call": "human_decision",
}

# A day of corrections, written the way a student actually types them.
CORRECTIONS = [
    {"text": "amount came as a string again", "cause": "bad_format", "area": "payments", "severity": 2},
    {"text": "paise not used, it sent rupees", "cause": "bad_format", "area": "payments", "severity": 2},
    {"text": "do not edit the README", "cause": "off_scope", "area": "docs", "severity": 1},
    {"text": "you forgot to install requirements", "cause": "setup_miss", "area": "env", "severity": 1},
    {"text": "the old login bug is back", "cause": "wrong_result", "area": "auth", "severity": 3},
    {"text": "README got changed once more", "cause": "off_scope", "area": "docs", "severity": 1},
]

def fingerprint(cause, area):
    """Same cause + same area = same control, whatever the wording."""
    return hashlib.sha1(f"{cause}|{area}".encode()).hexdigest()[:8]

def build_ratchet(corrections, promote_at=2):
    controls = {}
    for c in corrections:
        key = fingerprint(c["cause"], c["area"])
        control = controls.setdefault(key, {
            "id": key,
            "cause": c["cause"],
            "area": c["area"],
            "layer": LAYERS[c["cause"]],
            "count": 0,
            "severity": 0,
            "examples": [],
        })
        control["count"] += 1
        control["severity"] = max(control["severity"], c["severity"])
        control["examples"].append(c["text"])
    # Promote if it repeated, or if one hit was already costly.
    for control in controls.values():
        control["promote"] = control["count"] >= promote_at or control["severity"] >= 3
        control["review_in_days"] = 90
    return sorted(controls.values(), key=lambda x: -x["count"])

ratchet = build_ratchet(CORRECTIONS)
print(json.dumps(ratchet, indent=2))
print("\nAdd these controls now:")
for control in ratchet:
    if control["promote"]:
        print(f"  [{control['id']}] {control['area']}: add {control['layer']} (seen {control['count']}x)")

Look at the printed list. The two differently worded money complaints collapse into one control, because their fingerprint matches. The login bug is promoted after one report, because its severity is high. The setup miss stays on the watchlist, not in your rules file.

Where you will see this

• Coding agents like Claude Code and Cursor read a project instructions file, which is where promoted conventions end up.
• CI pipelines: every "this bug came back" becomes a regression test that blocks the merge.
• Customer-support bots: repeated wrong answers become fixed reply templates and knowledge-base entries.
• Swiggy or Zomato style assistants: repeated ordering mistakes become hard validation rules, not softer prompt wording.
• Any team's code review checklist, which is really a pile of past corrections nobody has automated yet.

Common mistakes

• Writing a rule that repeats the symptom instead of fixing the cause. The next similar case slips through untouched.
• Putting everything in the instructions file. Instructions are suggestions; tests and permissions are enforcement, and long files get ignored.
• Promoting every small preference. Your rules file grows into noise and the important lines get lost.
• Never deleting anything. Stale rules quietly fight your newer rules.
• Adding a control without verifying it. If you never watch the test fail once, you do not know it protects you.

If they ask in an interview

Q: How do you stop an AI coding agent from repeating the same mistake?

A: I treat each correction as evidence that some control is missing. I find the root cause, then move the lesson into the earliest layer that can prevent it — a test, a permission boundary, a setup script, or a canonical example. Chat corrections fix one run; a test fixes every run.

Q: Prompt instructions or tests — which one do you prefer, and why?

A: Tests, whenever the failure is checkable. An instruction is a request the model may or may not follow, while a failing test blocks the change outright. I use instructions only for conventions and judgment calls that cannot be expressed as an assertion.

Q: How do you keep your agent instructions from growing out of control?

A: I fingerprint each control by root cause and area so repeated reports merge instead of adding new lines. Every control gets an owner and a review date, and I delete it when the architecture changed or a stronger executable check replaced it.

Try these

1. Open your last long chat with any coding assistant. List every correction you typed twice, and write the real cause for each, not the symptom.
2. Take one of those causes and turn it into a failing test first. Watch it fail, then fix the code and watch it pass.
3. Extend the code above so severity 3 items print a suggested owner and a review window shorter than 90 days.
4. Feed the script two corrections with the same cause but different areas. Confirm they do not merge, then decide whether that separation suits your project.

Words, simply

Quick recap

• Every correction is information about a missing control, so capture it instead of repeating it.
• Fix the cause, at the earliest layer that can prevent it, and verify the control actually works.
• Merge repeated lessons by fingerprint and delete stale ones, so the system stays small and true.