# LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

arXiv: https://arxiv.org/abs/2605.12493 · HTML: https://mlsanity.com/paper/2605.12493/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 30 of 40)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents](https://mlsanity.com/paper/2602.02474/) (2026)
- [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://mlsanity.com/paper/2502.05167/) (2025)
- [MEMORYLLM: Towards Self-Updatable Large Language Models](https://mlsanity.com/paper/2402.04624/) (2024)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory](https://mlsanity.com/paper/2512.06688/) (2025)
- [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks](https://mlsanity.com/paper/2602.16313/) (2026)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs](https://mlsanity.com/paper/2510.27246/) (2025)
- [Mem-α: Learning Memory Construction via Reinforcement Learning](https://mlsanity.com/paper/2509.25911/) (2025)
- [AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications](https://mlsanity.com/paper/2602.22769/) (2026)
- [CL-bench: A Benchmark for Context Learning](https://mlsanity.com/paper/2602.03587/) (2026)
- [Sleep-time Compute: Beyond Inference Scaling at Test-time](https://mlsanity.com/paper/2504.13171/) (2025)

## Built on by (most cited first, 23 of 23)

- [TOKI: A Bitemporal Operator Algebra for Contradiction Resolution in LLM-Agent Persistent Memory](https://mlsanity.com/paper/2606.06240/) (2026)
- [From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agents](https://mlsanity.com/paper/2608.10502/) (2026)
- [GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation](https://mlsanity.com/paper/2606.22737/) (2026)
- [TEPA: Revoking Stale Memories for Conflict-Robust Language Agents](https://mlsanity.com/paper/2608.07429/) (2026)
- [Mi-Memory: A Lifecycle Memory Framework for Personal AI](https://mlsanity.com/paper/2607.18975/) (2026)
- [Supra Cognitive Modes: A Routed Architecture for Agent Memory](https://mlsanity.com/paper/2607.19096/) (2026)
- [MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use](https://mlsanity.com/paper/2608.20202/) (2026)
- [VoiceLongMemEval: Do Assistants Remember How You Sounded?](https://mlsanity.com/paper/2609.00570/) (2026)
- [Regimes: An Auditable, Held-Out-Gated Improvement Loop Demonstrated on LongMemEval with ActiveGraph](https://mlsanity.com/paper/2606.10241/) (2026)
- [Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents](https://mlsanity.com/paper/2607.10526/) (2026)
- [LOGOS: A Living Logic for AI Agent Teams That Evolve With Humans](https://mlsanity.com/paper/2607.10878/) (2026)
- [BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents](https://mlsanity.com/paper/2609.13149/) (2026)
- [Self-Evolving Search Index](https://mlsanity.com/paper/2609.19656/) (2026)
- [Correct Now, Insufficient Later: Auditing Update Sufficiency in Context Compression](https://mlsanity.com/paper/2609.20045/) (2026)
- [Contract Memory Compiler: Resolve, Then Traverse](https://mlsanity.com/paper/2609.32658/) (2026)
- [CADOC: Cache-Aware Dynamic Object Context for Long-Horizon Agents](https://mlsanity.com/paper/2609.37012/) (2026)
- [Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?](https://mlsanity.com/paper/2609.39578/) (2026)
- [Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval](https://mlsanity.com/paper/2610.02070/) (2026)
- [Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Models](https://mlsanity.com/paper/2610.05472/) (2026)
- [The Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory](https://mlsanity.com/paper/2610.07309/) (2026)
- [DSV-Mem: Evaluating Multimodal Memory in Professional Workflows for MLLM Agents](https://mlsanity.com/paper/2610.08102/) (2026)
- [MemoWM: How World Models Change What Agents Need to Remember](https://mlsanity.com/paper/2610.10778/) (2026)
- [VitaBench: Memory Lifecycles Beyond Retrieval](https://mlsanity.com/paper/2610.14731/) (2026)

## Abstract (arXiv)

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments. LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. Questions are paired with history trajectories containing up to 500 trajectories and 115M tokens. We use a context gathering formulation: memory systems consume history trajectories and return compact evidence for downstream question answering. We propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show that AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and the off-the-shelf coding agent baseline (69.3%). Despite the strong performance gains, coding agent based methods have high latency costs. While AgentRunbook-C advances the accuracy-latency Pareto frontier, substantial room for improvement remains. Together, these results establish LME-V2 as a challenging testbed for developing long-term memory systems for environment experience.