mlsanityData to 14 Oct 2026

MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader

arXiv 2610.05343 ↗First posted 4 Oct 2026cs.CL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

An adequate conversational answer may differ from a short or incomplete benchmark reference. To measure adequacy against the recorded history we prefer source-aware grading, in which the judge checks the reference against the full source before assessing system-blinded answers; original reference-only grading is reported alongside. With a local Qwen 3.8 27B Q4_K_M reader and a 24,000-token evidence ceiling, MemStrata CL1 scores 475/500 (95.0%) on LongMemEval-S and 1,400/1,540 (90.91%) on LoCoMo categories 1-4 under source-aware GPT-5.5 adjudication, against 463/500 (92.6%) and 1,205/1,540 (78.25%) under reference-only grading of the same answers. It preserves a retrieval backbone and adds nonduplicated, dated, speaker-attributed source spans. A same-reader full-history control with about 4.7 times the evidence scores 464/500 reference-only and 470/500 (94.0%) source-aware; neither difference is decisive. Keyword-only selection at the same budget scores 425, and a matched-reader Letta arm 438. On LongMemEval-M, where the packet holds about 1.6% of each history, MemStrata CL1 scores 427/500, with losses concentrated in multi-session and temporal questions. On 300 BEAM-1M questions it outscores dense retrieval, 0.738 to 0.706 (Wilcoxon p = 0.011). A same-seed replay of unchanged requests changed 1.5-2.3% of labels. On identical packets GLM 5.3 flash is non-inferior within 3 points (462 versus 463); Muse Spark 1.3 did not show non-inferiority on 269 questions. None of four pre-registered interventions met all of its registered advancement or feasibility criteria. Signed read-side artifacts support inspection but do not regenerate the private retrieval pipeline. The superiority of source-aware grading to human adjudication is not established, and development exposure, automated-judge dependence and the absence of held-out data preclude an independent-replication or leaderboard claim.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 18 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

10 of 18 references.

References · 10

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  4. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  5. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  6. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 400
  7. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  8. 2025Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMscited by 60
  9. 2026Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledgecited by 14
  10. 2026Temporal Validity on Real Software Histories: Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixescited by 2

Built on by

Not cited yet.