RESEARCH · AGENT MEMORY
How agents
remember.
An agent's context window is its whole conscious mind: everything it can hold at once, and nothing more. Real work does not fit. The interesting engineering is not the window's size, it is what an agent keeps, compresses, or writes down when the window fills. Get that wrong and the agent forgets its own goal halfway through the job.
The three tiers of agent memory: what stays in view, what gets compacted, what gets written down.
THE CONSTRAINT
The window is always too small.
A context window is finite and priced by the token. A long task, a multi-step tool run, a two-hour meeting: each produces more tokens than any window holds.
Keeping everything in view every turn is not just expensive, it actively hurts. A stuffed window pushes the important material into the middle, where the model half-ignores it (see the dumb zone). So agents cannot rely on the window alone. They need memory that lives outside it, and a discipline for moving information in and out.
THE TIERS
Three tiers of memory.
Context: the short term.
What the model sees this turn. Fast and precise, but finite and billed by the token. This is where the current step happens, and it should hold only what the current step needs.
Working memory: the scratchpad.
A running summary of the session, compacted as it grows. This is where "what have we decided so far" lives, so the agent can drop the raw transcript without losing the plot.
Long-term store: the record.
Files, notes, a vector index. Written once, retrieved on demand. This is where "what did we decide last week" lives, and it is why an agent can pick a task back up cold.
THE MOVES
Compaction is the whole game.
The window fills on every non-trivial task. What separates an agent that finishes from one that stalls is how it handles the overflow.
Summarize, do not truncate.
Dropping the oldest turns loses the decisions made in them. Summarizing those turns keeps the decisions and discards only the words.
Externalize state.
Write the plan, the open items, and the settled facts somewhere the agent can re-read. Then a full window is never a single point of failure.
Retrieve on demand.
Pull only the memory a step needs, when it needs it, instead of carrying the whole history forward every turn.
FAILURE MODES
How agents forget.
Two failures dominate, and both are silent.
Goal drift. The objective scrolls out of the window, and the agent starts optimizing the most recent instruction instead of the original one. It looks busy and productive while solving the wrong problem.
Context rot. A summary of a summary of a summary loses the one detail that mattered, and the agent proceeds confidently on a corrupted memory. Neither of these shows up as an error message. They show up as a wrong answer delivered with full confidence.
HOW MILLIE MEASURES THIS
Our multi-step cases are built to overflow the window on purpose. The scheduling-negotiation case runs long enough that a model must compact its own state to finish, and then we check whether the final booking still honors a constraint stated in the very first turn.
A model that only remembers the last thing you said cannot run a real workflow. We report which ones hold the thread. See what we measure.