Runtime Feedback Loops

In plain words

Suppose you ask a friend to run your code and he says 'working' without actually running it. Agents do the same thing. So you make every command go through one wrapper that saves what really happened: the output, the error text, the exit code and the time taken. The agent must read that saved record before it says anything. Exit code 0 means it worked. No exit code means stop, do not claim success.

How it flows

A tiny example

rec = run_with_feedback(["pytest"], note="expect 12 passed")
print(rec["exit_code"], rec["duration_ms"])

if rec["exit_code"] is None:
    stop("no exit code, cannot claim success")
elif rec["exit_code"] != 0:
    fix(rec["stderr_tail"])
else:
    move_on()

Notice the agent never decides from its own guess, only from the exit_code and stderr_tail sitting in the saved record.

What you will learn

• Why an agent that does not read real command output starts making things up.
• How to build a small feedback runner that saves what actually happened.
• How to cut huge outputs safely so the agent still sees the error.
• Why a missing exit code must stop the loop instead of passing it.

The problem, simply

See, think about your final year project demo. You tell your teammate Rahul, "Run the code and check." Rahul is busy on his phone and says "working, working." He never ran it. Next day the code crashes in front of the panel.

Rahul did not lie on purpose. He reported what he expected instead of what he saw.

Agents do exactly this. The agent writes "running the tests now," then "all tests passed." But maybe no test ran. Or it ran, failed, and the agent read only the first few lines and missed the failure at the bottom.

Our job is to remove that gap. Every command must come back as a small, honest record, and the agent must read that record before saying anything.

The idea

One runner, one record

Instead of letting the agent fire shell commands loosely, you make every command go through one wrapper function. Call it the feedback runner. It runs the command and writes a small record:

• command — the exact command as a list of words, so the shell cannot surprise you.
• stdout_tail and stderr_tail — the last few lines of normal output and of error output, kept separate.
• exit_code — 0 means success, anything else means failure. The one field you can trust.
• duration_ms — how long it took. Catches the test that silently hangs.
• started_at — when it started, so you can replay the run later.
• agent_note — one line where the agent writes what it expects to happen.

That last field is small but powerful. The agent writes "I expect all 12 tests to pass" before the command runs. The record then puts what really happened next to what was expected, so the mismatch is visible instead of hidden.

A worked example

Suppose Priya is building an agent that fixes bugs in a small Flask app. It edits a file, then wants to check its work.

Without the runner, it thinks "I fixed the import, tests should pass," and reports success.

With the runner, it calls run_with_feedback(["python3", "-m", "pytest"], "expect 12 passed"). The record comes back with exit_code: 1 and a stderr_tail ending in ModuleNotFoundError: No module named 'requests'. Exit code 1 is not 0, so the agent has no choice. It must read the real error and install the package.

Cutting output without losing the point

Some commands print 50 MB of logs. Paste all of that back to the model and you blow the context window and the cost.

So the runner trims — but not randomly. It keeps the first few lines and the last few lines, with a marker in between like ...truncated 8421 lines....

Two reasons for this shape. One, the same output always produces the same record, so you can replay and compare runs. Two, the useful part of a failing command — the final error, the summary line — sits at the tail. Keep only the first 10 lines and you throw away exactly what you needed.

Warning

Warning: Never use random sampling to shrink logs. Two runs of the same failing command would then produce different records, and you can no longer tell whether the code changed or the sampling changed.

Feedback is not the same as telemetry

You will hear the word telemetry — the general logging and tracing you set up for monitoring. It sounds like the same thing. It is not.

Feedback is for the next turn of this run; the agent reads it in seconds. Telemetry is for a human operator looking at hundreds of runs next month, asking "why did agents get slower this week?"

They share some fields but live in different files with different retention. Mix them, and your hot loop drags around months of history it does not need.

No exit code, no progress

Sometimes the runner itself breaks — the binary does not exist, or the process is killed. Then there is no exit code at all, and the record carries exit_code: null plus an error field saying why.

IMPRemember: when exit_code is null, the agent must refuse to claim success. No exit, no progress. A null is not a zero.

Three things that make it production-ready

Redact when you write, not when you read. Output can carry secrets — a Bearer token, password=, api_key=, an AWS or Slack key. Strip those lines before writing. Hide them only at display time and the raw secret still sits on disk, which is what an attacker reaches.

Rotate the file. Cap it at about 1 MB. On overflow, rename to .1, .2 and drop the oldest. The agent reads only the current file, so cost stays predictable — same idea as log rotation on any Linux server.

Link retries to their parent. Give every record a command_id, and give a retry a parent_command_id pointing at the failed attempt. Otherwise three failures then one success look like four independent successes, and the failure history vanishes.

Build it

"""A tiny feedback runner. Runs commands, saves honest records."""
import json, re, subprocess, time, uuid, os

RECORD_FILE = "feedback_record.jsonl"
KEEP_HEAD, KEEP_TAIL = 3, 5
SECRETS = re.compile(r"(Bearer |password=|api[_-]?key=|AKIA[0-9A-Z]{16})", re.I)

def redact(text):
    # Drop whole lines that look like they carry a secret. Do this BEFORE saving.
    lines = [("[REDACTED]" if SECRETS.search(ln) else ln) for ln in text.splitlines()]
    return lines

def trim(lines):
    # Deterministic: same output always gives the same trimmed result.
    if len(lines) <= KEEP_HEAD + KEEP_TAIL:
        return "\n".join(lines)
    cut = len(lines) - KEEP_HEAD - KEEP_TAIL
    kept = lines[:KEEP_HEAD] + ["...truncated %d lines..." % cut] + lines[-KEEP_TAIL:]
    return "\n".join(kept)

def run_with_feedback(command, agent_note, parent_id=None):
    started = time.time()
    record = {"command_id": uuid.uuid4().hex[:8], "parent_command_id": parent_id,
              "command": command, "agent_note": agent_note,
              "started_at": time.strftime("%H:%M:%S")}
    try:
        done = subprocess.run(command, capture_output=True, text=True, timeout=30)
        record["exit_code"] = done.returncode
        record["stdout_tail"] = trim(redact(done.stdout))
        record["stderr_tail"] = trim(redact(done.stderr))
    except Exception as err:                      # binary missing, timeout, killed
        record["exit_code"] = None                # null exit = the loop must stop
        record["error"] = type(err).__name__
    record["duration_ms"] = int((time.time() - started) * 1000)
    with open(RECORD_FILE, "a") as fh:
        fh.write(json.dumps(record) + "\n")
    return record

def agent_may_continue(record):
    if record["exit_code"] is None:
        return False, "no exit code captured, refusing to advance"
    if record["exit_code"] != 0:
        return False, "command failed, read stderr_tail before retrying"
    return True, "ok"

if __name__ == "__main__":
    jobs = [(["python3", "-c", "print('Bearer sk-secret-123'); print('12 passed')"], "expect a pass"),
            (["python3", "-c", "import sys; sys.stderr.write('ModuleNotFoundError\\n'); sys.exit(1)"], "expect a pass"),
            (["no_such_command_xyz"], "expect this to blow up")]
    for cmd, note in jobs:
        rec = run_with_feedback(cmd, note)
        ok, why = agent_may_continue(rec)
        print("exit=%s  %sms  continue=%s  (%s)" % (rec["exit_code"], rec["duration_ms"], ok, why))
        print("   stdout:", repr(rec.get("stdout_tail", "")), "\n   stderr:", repr(rec.get("stderr_tail", "")))
    os.remove(RECORD_FILE)

Look at three things. The Bearer line came back as [REDACTED], so the secret never touched the file. The middle command exits with 1, so continue=False even though the agent's note expected a pass. And the last command has exit=None, where the loop refuses to move on rather than guess.

Where you will see this

• Claude Code's Bash tool already captures stdout, stderr, exit code and duration for every command — the runner here is the same idea, written by hand.
• Cursor and similar coding assistants show terminal output inline, so the model reads what really happened, not a summary.
• CI pipelines store step logs as artifacts, so a reviewer can inspect a failed run days later without rerunning it.
• Support bots that call an internal refund API log the real status code, so "refund processed" can be checked against reality.

Common mistakes

• Letting the agent run commands outside the runner. One unlogged command brings guessing right back, because nobody can check that step.
• Treating a missing exit code as success. A crashed runner and a passing test are very different, but sloppy code turns both into "no error seen."
• Keeping only the first N lines of output. Errors and summaries print at the end, so head-only truncation drops the one line that mattered.
• Redacting secrets only when displaying. The raw token still sits on disk, which is exactly what leaks in a breach.
• Never rotating the record file. After a long session it becomes huge, and every loader call must read all of it.

If they ask in an interview

Q: Your agent claims the tests passed but they did not. How do you fix this at the system level?

A: Do not fix it with prompting. Route every command through a wrapper that records stdout, stderr, exit code and duration, and make the agent read that record before reporting. Treat a non-zero or missing exit code as a hard block on claiming success.

Q: What is the difference between runtime feedback and observability telemetry?

A: Feedback is for the agent's very next turn — short-lived, read within seconds. Telemetry is for humans reviewing many runs over weeks. They share fields but belong in separate files with separate retention.

Q: A command prints 50 MB of logs. How do you get it into the agent's context?

A: Keep a fixed number of lines from the head and the tail with a "truncated N lines" marker in between. It is deterministic, so the same output always gives the same record, and the tail is where the real error and summary live.

Try these

• Add a cwd field to each record, then run the same command from two folders and confirm the records are distinguishable.
• Make a fake string with a Slack-style token and an AWS-style key, and extend the redaction rule to catch both.
• Add rotation: when the file crosses 1 MB, rename it to .1 and start fresh, keeping three old files.
• Write a script that reads the file and prints only the most recent non-zero exit, with its command and stderr tail.

Words, simply

Quick recap

• An agent without real command output reports what it expected, not what happened. One runner recording every command closes that gap.
• Keep records small and deterministic — head plus tail with a marker — and strip secrets before writing, not while reading.
• Exit code 0 is the only success signal; a missing one means stop, and retries must point back at the attempt they came from.