mlsanityData to 14 Oct 2026

Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

arXiv 2607.21962 ↗First posted 24 Jul 2026cs.CLcited by 3

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We invert the pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction and separately validated for answerability. The synthetic, fictionalized corpus (~380 questions, 15 types) embeds features absent from the benchmarks we survey: per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets. Benchmarking five memory architectures against a no-memory control (fixed answerer, versioned LLM judge, three replicates, two horizons), we find backend rankings invert with history length: the budgeted curated-map memory that leads at three weeks loses recall of evicted content by nine weeks (96% to 72%) while a provenance-typed graph rises to 90%; the inversion is positive for all six users under complete cross-family re-judging (exact p=0.031). A full-rendered-history baseline ties or exceeds the best memory system at the short horizon but shows no judge-independent advantage at nine weeks, at about twice the read cost. Write-stage quality strongly correlates with downstream quality (weakly-written facts fail 24% vs 2%), and injection resistance tracked whether provenance boundaries survive representation. A layered architecture performs best among the memory systems in both regimes (96.8% short-horizon) and is released as Veracium, an open-source library, with the corpus generator and harness.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 50 of its 59 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

50 of 59 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation29 Jun
  2. 2026Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents25 Jun
  3. 2026Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge25 Jun
  4. 2026Are We Ready For An Agent-Native Memory System?23 Jun
  5. 2026Reliable Post-Retrieval Assembly for Agent Memory: Separating Evidence Extraction from Policy Execution31 May
  6. 2026MemGym: a Long-Horizon Memory Environment for LLM Agents20 May
  7. 2026Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory19 May
  8. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agents19 May
  9. 2026MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing4 May
  10. 2026The Missing Knowledge Layer in Cognitive Architectures for AI Agents13 Apr
  11. 2026Governing Evolving Memory in LLM Agents: Risks, Mechanisms, and the Stability and Safety Governed Memory (SSGM) Framework12 Mar
  12. 2026Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory2 Mar
9 more
  1. 2026Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations22 Feb
  2. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks18 Feb
  3. 2026Evaluating Memory Structure in LLM Agents11 Feb
  4. 2026Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents13 Jan
  5. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions7 Jul
  6. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  7. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  8. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memory20 Jan
  9. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2022Mass-Editing Memory in a Transformercited by 1,001
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  10. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  11. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
  12. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
17 more
  1. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  2. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  3. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  5. 2025Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMscited by 60
  6. 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agentscited by 33
  7. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
  8. 2026Hidden in Memory: Sleeper Memory Poisoning in LLM Agentscited by 27
  9. 2026Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guaranteescited by 18
  10. 2025Convomem Benchmark: Why Your First 150 Conversations Don't Need RAGcited by 10
  11. 2026MemAudit: Post-hoc Auditing of Poisoned Agent Memory via Causal Attribution and Structural Anomaly Detectioncited by 7
  12. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 6
  13. 2026SmartSearch: How Ranking Beats Structure for Conversational Memory Retrievalcited by 5
  14. 2026OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detectioncited by 5
  15. 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
  16. 2026Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agentscited by 3
  17. 2026Structured Episodic Event Memorycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Zero-Mem: Zero-Token Memory Operations for LLM Agents
  2. 2026MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
  3. 2026When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory
  4. 2026When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents
  5. 2025Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects
  6. 2026AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents
  7. 2026EdgeMem: LLM-Free Agent Memory Construction and Retrieval via Evidence-Preserving Multi-Anchor Hypergraph
  8. 2026What Makes Agent Memory Useful for Reliable Unanswerable Question Handling?