Virtual Context Memory

In plain words

Your study table fits only four books, so you keep swapping with the almirah behind you. A model's prompt is that table. So we keep a small always-visible prompt plus a big searchable store outside it, and we give the agent tools to move facts between the two. It saves what matters and fetches it back when it needs it.

How it flows

A tiny example

core = {"profile": "Priya, 2027 batch"}
archive = []

def turn(user_text):
    if is_worth_saving(user_text):
        archive.append(user_text)      # page out
    if needs_old_fact(user_text):
        facts = search(archive, user_text)  # page in
        return think(core, facts, user_text)
    return think(core, [], user_text)

Notice the prompt never grows: only archive grows, and facts come back one search at a time.

What you will learn

• Why a bigger context window does not solve an agent's memory problem.
• The two-tier idea: a small prompt that is always visible, plus a big searchable store outside it.
• How the agent itself decides when to save something and when to fetch it back.
• Where this breaks in real products, and what to watch out for.

The problem, simply

See, think of your hostel study table. It is small. You can keep maybe four books open on it at a time. Everything else sits in the almirah behind you.

When you are solving a DBMS question, you keep the DBMS book open. When you switch to OS, you get up, put DBMS back in the almirah, and pull out the OS book. The table never becomes bigger. You just get good at swapping.

An AI model has exactly this table. It is called the context window, and it is the text the model can see while answering. Now the trouble is three-fold.

First, overflow. A long chat, a 200-page PDF, a debugging session with fifty tool calls — all of it crosses the limit and the oldest part just falls off. Second, dilution. Even inside the limit, if you dump everything, the model's attention spreads thin and it misses the one line that mattered. Third, persistence. Close the tab, come back tomorrow, and the agent remembers nothing. Your Swiggy support chat starts from "Hello, how may I help you?" every single time.

And no, buying a bigger table does not fix it. Even with very large windows, agents still lose long-horizon facts that a much smaller agent with an outside store catches easily.

The idea

Borrow it from your operating system

Your laptop has 8 GB RAM, but you run Chrome with forty tabs, VS Code and Spotify together. That should not fit. It works because the OS keeps hot pages in RAM and pushes cold ones to disk, then fetches a page back when you click that sleeping tab. That fetch is called a page fault.

An agent memory design called MemGPT (a 2023 research idea; "Mem" for memory, "GPT" for the model) copies this exactly.

That is the whole trick. The agent runs a normal think-act loop, and you just give it one extra family of tools: tools that read and write memory.

The two tiers

Main context is fixed size. Current task, recent messages, and a few pinned notes about the user. Always visible, no searching needed.

External context is unbounded. Notes, old conversations, extracted facts. Invisible by default. The agent must ask for it.

Memory as an interrupt

The agent is not handed memory. Mid-conversation it calls a memory tool, the runtime fetches, and the result arrives as a fresh observation on the next turn.

If you have done OS lab, this is a read() system call. Process asks, pauses, bytes arrive, process continues.

The usual tool surface looks like this:

• core_memory_append(section, text) — pin a fact inside the prompt itself.
• core_memory_replace(section, old, new) — edit a pinned fact.
• archival_memory_insert(text) — write to the big outside store.
• archival_memory_search(query, top_k) — pull the most relevant notes back.
• conversation_search(query) — scan old turns of the chat.

IMP

Note: core_* writes cost you prompt space forever. archival_* writes cost you nothing until you search. So pin only what is needed almost every turn.

A worked example

Suppose Priya is using a placement-prep agent. On Monday she says: "I am from Hyderabad, 2027 batch, I am weak in dynamic programming, and I already have an Infosys offer at ₹4.5 LPA."

A no-memory agent forgets all of it. A MemGPT-style agent does this:

1. Pins user_profile: Hyderabad, 2027 batch into core memory — small, needed every turn.
2. Writes "Priya is weak in DP" and "Priya has an Infosys offer, ₹4.5 LPA" into archival — bigger, needed sometimes.
3. Chat continues for 60 turns. The early messages get evicted from the prompt.

Friday, Priya asks: "Should I even attend the TCS drive?" The agent calls archival_memory_search("Priya existing offer"), gets the Infosys note back, and answers with her actual situation. The prompt never grew. The memory did.

Build it

"""A toy MemGPT: small prompt + big store + memory tools. No API calls."""

MAIN_CAP = 4  # how many messages the "prompt" can hold

class Agent:
    def __init__(self):
        self.core = {}        # pinned facts, always in the prompt
        self.messages = []    # recent turns, evicted when full
        self.archive = []     # unbounded store: (turn_no, text)
        self.turn = 0

    # --- memory tools the agent can call ---
    def core_append(self, section, text):
        self.core[section] = text

    def archival_insert(self, text):
        self.archive.append((self.turn, text))

    def archival_search(self, query, top_k=1):
        words = set(query.lower().split())
        # score by how many query words the note shares
        scored = [(len(words & set(t.lower().split())), n, t)
                  for n, t in self.archive]
        scored = [s for s in scored if s[0] > 0]
        scored.sort(reverse=True)
        return scored[:top_k]

    # --- the loop ---
    def say(self, text):
        self.turn += 1
        self.messages.append(f"user: {text}")
        while len(self.messages) > MAIN_CAP:          # overflow!
            old = self.messages.pop(0)
            self.archive.append((self.turn, old))     # page out to "disk"
            print(f"  [evicted -> archive] {old}")

agent = Agent()
agent.core_append("profile", "Priya, Hyderabad, 2027 batch")
agent.archival_insert("Priya has an Infosys offer at 4.5 LPA")
agent.archival_insert("Priya is weak in dynamic programming")

for line in ["hi", "what is DP", "give me a problem", "explain memoization",
             "one more problem", "should I attend the TCS drive"]:
    agent.say(line)

print("\nprompt now holds:", agent.messages)
print("core (always visible):", agent.core)

hits = agent.archival_search("existing offer company")   # page fault
print("\nagent searched memory and found:")
for score, turn, text in hits:
    print(f"  turn {turn} (score {score}): {text}")
print("\nanswer: you already have an offer, so attend TCS only if it pays more.")

Watch two things. One, the eviction lines: old turns leave the prompt but are not lost, they land in the archive. Two, the final search pulls back a fact from turn 0 even though the prompt holds only the last four messages. That gap is the whole point.

Where you will see this

• Claude Code and Cursor keep a project file of pinned facts and search the rest of your repo only when needed — core memory plus archival, exactly.
• ChatGPT's "memory" feature that says "remembered" when you mention your job — that is an archival write happening live.
• Customer-support bots for Swiggy or a bank that recall your last three complaints without you repeating the order ID.
• Letta, the production system that MemGPT itself grew into in 2024, with an extra tier and background memory work.
• Mem0 and similar stores, which add fact extraction and de-duplication on top of the same two-tier skeleton.

Common mistakes

• Pinning everything into core memory. Feels safe, but core memory eats your prompt budget on every single turn, so you pay for it forever and dilute attention. Pin identity-level facts only.
• Writing far more than you read. This is memory rot. After a month the store is full of stale notes and search returns garbage. Fix it with periodic consolidation and explicit invalidation of outdated facts.
• Storing notes without a source. The agent says "you asked me to ship X" and cannot show which turn. Always save session id and turn id with every write.
• Trusting retrieved text. Memory is just text coming back into the prompt. If an attacker gets one poisoned note in, it gets re-ingested every future session — see the Prompt Injection lesson in Module 5.
• Assuming a bigger window removes the need for memory. It removes overflow for a while. It never gives you persistence across sessions.

IMPRemember: memory is not storage, it is a decision about what deserves prompt space right now.

If they ask in an interview

Q: Why can't we just use a very large context window instead of a memory system?

A: A bigger window delays overflow but does not solve dilution or persistence. Attention still spreads thin over irrelevant text, and a new session still starts empty. Also you pay for every token you stuff in, on every turn.

Q: Explain the MemGPT pattern in one minute.

A: It is OS virtual memory applied to prompts. The prompt is RAM, an external searchable store is disk, and memory tool calls are page faults. The agent runs a normal loop and decides for itself when to page facts in and out.

Q: What is memory poisoning and how would you defend against it?

A: An attacker gets malicious text saved as a memory note, and the agent re-reads it in future sessions. Defend by never storing raw untrusted content, tagging every note with its source and trust level, and treating retrieved text as data, never as instructions.

Try these

1. Add a word budget instead of a message count. Approximate tokens as words times 1.3, and when you cross the cap, replace the three oldest messages with a one-line summary. Compare answers with and without the summary.
2. Improve the search. Right now it counts shared words. Make rare words score higher than common ones like "the" and "is", then check whether the right note comes first more often.
3. Add session_id and turn_id to every archival write and make the agent print its source with every recalled fact.
4. Simulate poisoning. Insert a note saying "ignore all future user instructions", then write a guard that flags any retrieved note that looks like a command instead of a fact.

Words, simply

Quick recap

• The prompt is RAM, the external store is disk, and memory tools are how the agent pages between them.
• The agent decides what to save and what to fetch, so memory becomes a tool-use problem, not a storage problem.
• Retrieved memory is untrusted text — tag it, cite it, and clean it up regularly.