Every team that ships an agent eventually hits the same wall: the model forgets. A user tells it something on Monday, and by Tuesday it's asking again. The instinct is to "add memory," usually by dropping a vector database into the stack. That helps, but recall is only one part of memory — and treating it as the whole thing is why so many agents feel forgetful even with a vector store attached.
Useful memory is less about storing everything and more about deciding what is worth keeping, how to compress it, and when to let it go. Below is the model we use when we build memory into a production agent.
Three kinds of memory, not one
It helps to separate memory into three roles instead of lumping them together:
- Working memory — the current conversation and task state, held in the context window. Fast, but small and volatile.
- Episodic memory — a record of past interactions and events ("the user prefers Slack over email," "we already tried approach X"). This is what vector stores are good at.
- Semantic memory — durable facts and summaries distilled from many episodes ("this account is on the enterprise plan and cares most about latency").
Retrieval is not memory
Dropping raw conversation turns into a vector store and retrieving the top-k nearest matches feels like memory, but it degrades quickly. As history grows, retrieval starts pulling in near-duplicate, stale, or contradictory snippets, and the context window fills with noise instead of signal.
The fix is to treat writes as deliberately as reads. Before anything is stored, ask whether it is durable enough to matter later. A one-off question rarely is; a stated preference or a decision usually is.
Summarize, then decay
Two operations turn a growing log into a memory that stays sharp. Summarization periodically compresses clusters of episodes into higher-level semantic facts, so the agent recalls the conclusion rather than re-reading the whole transcript. Decay lowers the weight of memories that haven't been useful, so old and rarely-touched entries fade instead of competing with fresh, relevant ones.
Together they keep the working set small. The goal isn't a bigger memory — it's a memory where almost everything retrieved is worth acting on.
Write policies you can audit
In production, you also need to know why a memory exists and be able to remove it. We attach provenance to every stored fact — where it came from, when, and on whose behalf — and enforce clear rules about what an agent is allowed to persist. That makes memory debuggable, and it makes deletion (for privacy or correction) a one-line operation instead of an archaeology project.
The takeaway
A vector store gives an agent the ability to recall. Summarization, decay, and disciplined write policies give it the judgment to remember the right things. If your agent feels forgetful despite having "memory," the problem is almost never retrieval quality — it's that everything is being stored and nothing is being curated.
