The Feedback Ratchet

In plain words

Think of the hostel mess complaint register. Complaints get written every week and nothing changes, because no one owns a specific fix. A feedback ratchet is the opposite: every signal from your agent goes to the layer that can actually stop it, gets one owner and one proof, and later gets reviewed and retired. Forward only, never slipping back.

How it flows

A tiny example

signal = observe()
layer = route(signal)          # test, context, policy, runtime
action = {
    "owner": "Priya",
    "proof": "test blocks a Rs 500 refund",
    "retire_when": "payments API enforces the cap",
}
apply(layer, action)
verify(action["proof"])

Notice that owner, proof and retirement are written at the same moment as the fix, not added later.

What you will learn

• Why collecting feedback is useless until it changes something in the system.
• How to send each complaint or failure to the right layer that can actually fix it.
• How to give every fix an owner, a proof, and an expiry date.
• How to remove old rules that no longer earn their place.

The problem, simply

Think about your hostel mess complaint register. Every week students write in it. "Dal was watery." "Milk got over by 8 am." "Chapati count reduced."

The register fills up. Nobody argues that the complaints are wrong. But next month the same lines get written again, because nobody was put in charge of one specific change with one specific check.

Now run it properly. "Milk got over by 8 am" goes to the person who orders supplies, not to the cook. He raises the morning order by twenty litres. Next week somebody checks at 8 am whether milk is still there. That is one complaint that permanently changed the system.

Your agent is the same. Users complain, tests fail, an incident happens at 2 am, someone corrects the agent's code. If all of that just sits in a dashboard, you have a complaint register. A ratchet makes each signal push the system one notch forward, and never let it slip back.

IMP

Note: A ratchet is that toothed wheel in a cycle's rear hub. It turns forward freely, but it cannot turn back. That is the whole idea in one word.

The idea

From signal to durable change

Every useful piece of feedback goes through the same six moves.

Most teams collect signals and never review. The middle is where the learning actually happens.

Route it to the layer that owns the cause

When the agent does something silly, the reflex is to add one more line to the prompt. "Please do not delete files."

But a prompt is a request. A test is a proof. A permission check is a wall. Fix the cause at the earliest layer that can actually stop it. Memorise this table.

An owner and an expiry, always

Suppose Priya is building a support agent for a food delivery app. Three signals land on Monday.

One: the agent refunded ₹450 to a customer without any approval. That is an authority gap, so it goes to the policy layer. Owner: Priya. Change: refunds above ₹200 require a human click. Proof: a test that tries a ₹500 refund and expects a block.

Two: the agent told a user the order was delivered when it was not. Wrong result, so it goes to the evaluation layer. Owner: Rahul. Change: save that exact conversation as a test case. Proof: the test suite fails today and passes after the fix.

Three: the payment service timed out and the agent retried eleven times. Runtime problem. Owner: Sneha. Change: three retries with a backoff, then a clean error message.

Each one has a person, a specific artifact to edit, and something you can run to prove it worked.

IMPRemember: An improvement without an owner and a verification is not an improvement. It is a note.

Retirement is part of the design

Six months later Priya's system has forty policy rules. Some fight each other. Some block work that is perfectly fine, and nobody remembers why rule 19 exists.

So every control gets a review date and a retirement condition written the moment you add it. Retire it when the architecture changed, when a lower-level check now covers it anyway, when the failure it guards never appeared in the review window, or when it blocks honest work more often than it stops harm.

But retire on evidence, not on feeling. "This rule looks old" is not a reason. "This rule blocked forty valid refunds and zero fraud attempts in three months" is.

Both tracks use the same ratchet

Product feedback from real users changes what you build and how much of it. Corrections you give a coding agent change your tests, your context files, and your scope. A serious incident usually changes both.

That is why shaping the product is not a phase that finishes before coding starts. It keeps running through every accepted change.

Build it

#!/usr/bin/env python3
"""A tiny feedback ratchet: route signals, own them, rank them."""

import json

# Which layer owns which kind of failure. Earliest effective layer wins.
ROUTES = {
    "wrong_result": "evaluation",
    "missing_fact": "context",
    "unsafe_action": "policy",
    "timeout": "runtime",
    "new_need": "backlog",
}

# How bad is it if this happens again?
SEVERITY = {"low": 1, "medium": 3, "high": 9}

def make_action(signal):
    """Turn one raw signal into an owned ratchet action."""
    layer = ROUTES.get(signal["kind"], "backlog")
    # Priority = how bad it is, multiplied by how often we have seen it.
    score = SEVERITY[signal["severity"]] * signal["times_seen"]
    return {
        "what": signal["what"],
        "layer": layer,
        "owner": signal["owner"],
        "score": score,
        "proof": signal["proof"],
        "retire_when": signal["retire_when"],
    }

signals = [
    {"kind": "unsafe_action", "what": "Refunded Rs 450 with no approval",
     "severity": "high", "times_seen": 2, "owner": "Priya",
     "proof": "test: Rs 500 refund is blocked",
     "retire_when": "payments service enforces the cap itself"},
    {"kind": "wrong_result", "what": "Said 'delivered' for an undelivered order",
     "severity": "high", "times_seen": 4, "owner": "Rahul",
     "proof": "saved chat replays green in the eval suite",
     "retire_when": "never - keep the test case forever"},
    {"kind": "timeout", "what": "Retried the payment API 11 times",
     "severity": "medium", "times_seen": 3, "owner": "Sneha",
     "proof": "log shows at most 3 retries",
     "retire_when": "the API publishes a 99.9 percent uptime promise"},
    {"kind": "new_need", "what": "Users want order history in Telugu",
     "severity": "low", "times_seen": 6, "owner": "Arjun",
     "proof": "shipped slice gets used by 50 users",
     "retire_when": "not a control - it is a feature"},
]

backlog = sorted((make_action(s) for s in signals),
                 key=lambda a: a["score"], reverse=True)

for i, action in enumerate(backlog, 1):
    print(f"{i}. [{action['layer']:>10}] score={action['score']:<3} "
          f"owner={action['owner']}")
    print(f"    {action['what']}")
    print(f"    proof : {action['proof']}")
    print(f"    retire: {action['retire_when']}\n")

print(json.dumps({"open_actions": len(backlog)}))

Look at the layer column first. The refund problem went to policy, not to the prompt, and the retry storm went to runtime, not to the backlog. Then look at the order: the "delivered" lie ranks above the refund even though both are high severity, because it happened four times instead of two. Try adding a missing_fact signal and check that it lands on context.

Where you will see this

• Claude Code and Cursor keep project instruction files that grow every time you correct the agent. Each correction is a ratchet notch.
• Support bots at banks and telecoms keep a suite of past bad conversations as permanent test cases, so an old mistake cannot come back.
• Swiggy and Zomato style assistants put hard money limits in the payment service itself, not in the model's instructions.
• Post-incident reviews end with owned action items and due dates, which is the same six-step loop.
• GitHub Copilot style tools track which suggestions people accept or reject, and that changes ranking, not just a dashboard.

Common mistakes

• Adding a prompt line when a test or permission would do. A prompt is a polite request that the model may ignore under pressure. A permission check cannot be ignored.
• Collecting signals with no owner. Everybody nods in the meeting, nobody edits a file, and the same failure shows up next month.
• Skipping verification. If you cannot run something that fails before the fix and passes after it, you do not actually know the failure is less likely now.
• Never retiring anything. Rules pile up, start contradicting each other, and slow the agent down until people work around the whole system.
• Ranking by loudest complaint. One angry user is not the same as a failure that happened forty times quietly. Rank by consequence times frequency.

If they ask in an interview

Q: Your agent gave a wrong answer in production. Walk me through what you do.

A: First I save that exact input as a reproducible test case, so the failure is captured before I forget it. Then I find the earliest layer that owns the cause, which for a wrong result is usually evaluation or context rather than the prompt. Finally I assign one owner and a check that proves the fix, so we know it stopped happening.

Q: How do you stop your agent's rule list from becoming a mess over time?

A: Every control gets a review date and a written retirement condition on the day it is added. At review I check evidence: has the failure it guards appeared in the window, and is it blocking more legitimate work than harm it prevents. If a deeper check now covers it, the higher-level rule goes.

Q: Why not just put every fix in the system prompt? It is faster.

A: Because a prompt only asks the model to behave, while a test proves behaviour and a permission boundary makes the bad action impossible. Prompt-only fixes also compound into a huge instruction file that nobody can reason about, and long instructions get followed less reliably.

Try these

1. Take three real complaints from any app you use daily and write down which of the five layers each one belongs to, and why.
2. Extend the code so each action also carries a review_date, and print any action whose date has passed.
3. Add a fifth signal type of your own, route it, and check that the priority order changes the way you expected.
4. Pick one rule from a project you have worked on and write an honest retirement condition for it in one sentence.

Words, simply

Quick recap

• Feedback only counts when it reaches a durable change with an owner and a proof; otherwise it is a complaint register.
• Fix the cause at the earliest layer that can stop it: test, context, permission or runtime, before touching the prompt.
• Write the retirement condition on the day you add the control, and retire on evidence, not on feeling.