Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks
Not on the map. Why?
Abstract (arXiv)
Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index. This raises an underspecified evaluation question: which stored form should receive retrieval credit? We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions. We present TIAP, a fixed-output audit that rescores saved ranked outputs under three targets -- Raw, Source, and Canonical -- without rerunning retrieval. On LoCoMo and LongMemEval-S, switching only the credited target changes nDCG on 83.4--94.0 percent of shared queries, flips target orderings on Mem0 and MemoryOS transfer runs, and reverses parser-density recommendations. A 1,902-case semantic audit further shows that relaxed source-linked credit is fully justified only 29.2 percent of the time, despite high rubric reliability in a validation subset. These results reveal target noninvariance: conclusions about memory architectures can silently flip with a single benchmark-design choice. Conversational-memory papers should therefore define and report the scoring target explicitly.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2026.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Fixed RAG Compression Collapses Measured Reader Scalingcited by 1
- 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
- 2026When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Modelcited by 0
- 2026Memory Consolidation Flattens the Temporal Shape of User Factscited by 0
- 2026Memory Consolidation Flattens the Temporal Shape of User Facts29 Sep 2026
- 2026When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model28 Sep 2026
- 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model14 Aug 2026
- 2026Fixed RAG Compression Collapses Measured Reader Scaling20 Jun 2026