# What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory

arXiv: https://arxiv.org/abs/2609.08279 · HTML: https://mlsanity.com/paper/2609.08279/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 26 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 25 of 25)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [ContextCite: Attributing Model Generation to Context](https://mlsanity.com/paper/2409.00729/) (2024)
- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.13781/) (2024)
- [Causal Estimation of Memorisation Profiles](https://mlsanity.com/paper/2406.04327/) (2024)
- [Forgetful but Faithful: A Cognitive Memory Architecture and Benchmark for Privacy-Aware Generative Agents](https://mlsanity.com/paper/2512.12856/) (2025)
- [Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory](https://mlsanity.com/paper/2602.06025/) (2026)
- [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://mlsanity.com/paper/2605.09649/) (2026)
- [Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems](https://mlsanity.com/paper/2605.26302/) (2026)
- [The Missing Memory Hierarchy: Demand Paging for LLM Context Windows](https://mlsanity.com/paper/2603.09023/) (2026)
- [Beyond Compaction: Structured Context Eviction for Long-Horizon Agents](https://mlsanity.com/paper/2606.11213/) (2026)
- [EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agents](https://mlsanity.com/paper/2606.05894/) (2026)
- [Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory](https://mlsanity.com/paper/2603.02473/) (2026)
- [SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval](https://mlsanity.com/paper/2603.15599/) (2026)
- [Learning What to Remember: Observability-Safe Memory Retention via Constrained Optimization for Long-Horizon Language Agents](https://mlsanity.com/paper/2606.10616/) (2026)
- [MemFail: Stress-Testing Failure Modes of LLM Memory Systems](https://mlsanity.com/paper/2605.26667/) (2026)
- [Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents](https://mlsanity.com/paper/2606.11680/) (2026)
- [WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems](https://mlsanity.com/paper/2605.24579/) (2026)
- [REAL: A Reasoning-Enhanced Graph Framework for Long-Term Memory Management of LLMs](https://mlsanity.com/paper/2606.10694/) (2026)

## Abstract (arXiv)

Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the resulting loss in accuracy, but do not distinguish irreversible losses caused by eviction from recoverable retrieval failures. We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context and reruns the same reader. Combining the change in correctness with whether the evidence was retained after eviction classifies each oracle-answerable error as recoverable, irreversible, or residual; in the residual case, the answer remains incorrect after restoration. We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes, using GPT-4o-mini as the primary reader and judge and GPT-5.4-mini as a robustness reader. Under top-k retrieval at an 80k-token budget, the irreversible share among errors corrected by restoration is 0.67-0.73 for FIFO, random, and redundancy-aware eviction, compared with 0.60 for LLM-importance. At 8k tokens, it reaches 1.00 for all four policies. Recoverable errors occur under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not directly comparable unless the retrieval regime is reported. An exploratory matched-accuracy analysis detects no difference in irreversible rate among accuracy-matched policy pairs at a resolution of 1.2-6 percentage points. The same analysis detects the deliberately destructive control. To our knowledge, this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores on a standard conversational benchmark.