Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
Not on the map. Why?
Abstract (arXiv)
A language model's memory can be worse than no memory at all when the model or its interface is disposed to act on it: a memory that keeps a wrong conclusion but drops the work behind it leads a model to re-emit the stale value as a confident answer, where an empty memory leads it to abstain. We call this brittle memory. The information loss is definitional; the finding is behavioral, and it turns on one thing, whether the memory kept a re-derivation basis (the source) rather than the answer. We measure it with reclaim evaluation: induce a known drift, compress at a fixed budget, deliver a correction that names the error, and score exact recovery, judge-free. Holding the budget fixed and varying only what the compression keeps isolates correctability from capability and from size; an 8B model and a frontier one wall in the same place. A one-line source-first policy, keep the recomputable source, drop the re-derivable conclusion, restores correctability at equal budget where the source is compact and identifiable, with a length-matched control ruling out "more text." We map where the fix fails, show the failure compounds through memory loops, and replicate across three deployed memory systems, real dialogue (MultiWOZ), and tau-bench, a deployed-agent benchmark where whether a lossy memory becomes a harmful action is a joint model-and-interface property. We release the harness, the paired memory conditions, and validators built to come out false.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 24 Jun 2026.
Builds on
26 of 33 references.
References · 26
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
14 more
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
- 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
- 2021Fast Model Editing at Scalecited by 498
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
- 2023Learning to Compress Prompts with Gist Tokenscited by 390
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
- 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
- 2023Adapting Language Models to Compress Contextscited by 356
- 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
- 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
- 2023Do Large Language Models Know What They Don't Know?cited by 243
- 2026When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Modelscited by 3
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Factscited by 3
- 2026They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On Itcited by 2
- 2026CrystalMem: Elastic Memory for Self-Evolving LLM Agents via Knowledge Crystallizationcited by 2
- 2026FACTWASH: Catching AI Rewrites That Wash Hearsay into Factcited by 1
- 2026What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metricscited by 0
- 2026Re-derivability Decides What a Staged Agent Pipeline Recovers After an Upstream Faultcited by 0
- 2026The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfacescited by 0
- 2026The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfaces26 Sep 2026
- 2026Re-derivability Decides What a Staged Agent Pipeline Recovers After an Upstream Fault26 Sep 2026
- 2026What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics17 Aug 2026
- 2026FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact4 Aug 2026
- 2026CrystalMem: Elastic Memory for Self-Evolving LLM Agents via Knowledge Crystallization31 Jul 2026
- 2026They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It3 Jul 2026
- 2026Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts28 Jun 2026