Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver
Not on the map. Why?
Abstract (arXiv)
MemoryAgentBench's Conflict Resolution split is read as measuring "selective forgetting". We execute the benchmark's own rule - the newest statement about a fact wins - as a zero-learning resolver frozen on one of the four fact lists. Under the official metric the rule answers 80.25% of the questions (74.5% on the three held-out lists). Of the rest, 67 items have a released gold that the last-write graph cannot reach but overwritten statements would ("The capital of India is New Delhi." superseded by "The capital of India is Grosseto."; gold New Delhi); such items are a third of the multi-hop questions at 262K. Two long-context models and our pre-registered approximate re-implementation of the benchmark's BM25 agent, one retained run per item and outcomes only, score 84.7%, 82.6% and 41.6% on the items the rule solves against 10.4%, 11.9% and 6.0% on those 67. The failures are a reachability split plus a small parser-scope residual; the per-item split, not the aggregate, is the unit at which a score here can be read.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 35 references that are in the map. First posted on arXiv 6 Oct 2026.
Builds on
15 of 35 references.
References · 15
- 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
- 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
- 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
- 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
- 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
- 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
- 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 34
- 2026Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitationscited by 21
- 2026On Randomness in Agentic Evalscited by 17
- 2026Are We Ready For An Agent-Native Memory System?cited by 15
3 more
- 2026Log analysis is necessary for credible evaluation of AI agentscited by 10
- 2026How to Interpret Agent Behaviorcited by 10
- 2026Can Agent Memory Systems Track Evolving State?cited by 7
Built on by
Not cited yet.