Causal Memory Selection
Select memories by their measured effect on the answer. A memory belongs in context only if it improves the next step.
The Problem This Solves
Memory retrieval usually starts with similarity. The agent embeds the current query, finds nearby memories, and adds the top matches to context.
Similarity is a weak proxy for usefulness. A memory can be topically related and still distract the agent, repeat stale assumptions, or pull the answer toward an old state of the world. Persistent memory makes this worse because old decisions accumulate and keep looking relevant.
The question is not: does this memory match the query? The question is: does this memory improve the next answer?
How It Works
Test candidate memories by their effect. For each memory, compare the agent’s answer with and without that memory, or use a cheaper evaluator that estimates whether the memory changes the next step in a useful direction.
The selected set should pass three tests before it reaches the context window:
| Test | Meaning |
|---|---|
| Relevance | The memory applies to the current task, user, or artifact |
| Helpfulness | Including it improves the expected answer or action |
| Non-harm | It does not introduce stale, unsafe, or misleading influence |
Causal Intervention-Based Memory Selection formalizes this idea. It evaluates candidate memories under controlled interventions and keeps memories that causally improve the response. The paper’s Causal-LoCoMo setup includes useful memories, distractors, and harmful memories, which is exactly the pressure persistent agents face in practice.
You do not need the full research method to use the pattern. The engineering move is simpler: add a selection gate after retrieval, then judge memories by downstream effect rather than retrieval score alone.
Example
A support agent is helping a customer renew a contract. Memory search returns:
| Memory | Similarity | Effect |
|---|---|---|
| Customer prefers annual billing | High | Useful |
| Customer complained about invoice wording in 2023 | High | Probably irrelevant |
| Customer’s old plan included seats that no longer exist | High | Harmful if used |
| Customer requires procurement approval above 10k | Medium | Useful |
A similarity-only retriever may include the first three. A causal selection gate keeps annual billing and procurement approval, then drops the invoice complaint and obsolete plan detail.
The final context is less complete, but the answer is better because the obsolete memory never gets a chance to steer the renewal path.
When to Use
- Long-lived agents with persistent user, project, or organization memory
- Retrieval systems where stale memories often look semantically relevant
- Domains where the wrong remembered fact can change the action
- Memory stores that include summaries, reflections, preferences, or previous decisions
When Not to Use
- Stateless RAG over authoritative documents
- Small memory stores where humans manually curate every item
- Cases where recall matters more than action quality, such as legal discovery or audit export
- Ultra-low-latency paths where every extra evaluation call is too expensive
Related Patterns
- Retrieval as Context Curation is the broader retrieval discipline this pattern sharpens
- Select, Don’t Dump defines the standard: only useful context earns space
- Temporal Decay handles age; causal selection handles effect
- Context Poisoning explains why harmful memories must be filtered before they become working context