mlsanityData to 7 Oct 2026

When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory

arXiv 2605.07313 ↗First posted 8 May 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Memory-agent evaluations report fixed-snapshot accuracy or retrieval quality, but these scores do not show whether evidence remains usable as irrelevant sessions (sessions not annotated as task-relevant evidence for the query) accumulate. We present a scale-conditioned evaluation protocol for agent memory under evidence-preserving growth: for each query, task evidence is held fixed while irrelevant sessions are added. The protocol logs agent--memory trajectories and reports four diagnostics: budget-compliant reliability, tail memory-call burden, failure-regime decomposition, and the usable-scale boundary where reliability falls below the target. Applied to LongMemEval and LoCoMo across flat, planar, and hierarchical memory interfaces, the protocol shows reliability loss is not a single phenomenon. On LongMemEval, HippoRAG stays within the two-call budget but loses 16--20 percentage points in budget-compliant reliability as irrelevant sessions are added; LiCoMemory's observed failures depend strongly on the agent, with Qwen3-8B exceeding the budget while Qwen3-32B and Qwen3-235B remain reliable in the tested range. The result supports a framework for making scalable-memory claims conditional on agent, interface, scale range, and interaction budget.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 24 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

23 of 24 references.

References · 23

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  12. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
11 more
  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  2. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  4. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  5. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  6. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  7. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  8. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 72
  9. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 68
  10. 2025LiCoMemory: Lightweight and Cognitive Agentic Memory for Efficient Long-Term Reasoningcited by 37
  11. 2026Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitationscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
  2. 2026The Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory Agentscited by 1