mlsanityData to 14 Oct 2026

Beyond the Needle's Illusion: Decoupled Evaluation of Evidence Access and Use under Semantic Interference at 326M-Token Scale

arXiv 2601.20276 ↗First posted 28 Jan 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Long-context LLM agents must access the right evidence from large environments and use it faithfully. However, the popular Needle-in-a-Haystack (NIAH) evaluation mostly measures benign span localization. The needle is near-unique, and the haystack is largely irrelevant. We introduce EverMemBench-S (EMB-S), an adversarial NIAH-style benchmark built on a 326M-token MemoryBank. While the full MemoryBank spans 326M tokens for retrieval-based (RAG) evaluation, we evaluate native long-context models only at scales that fit within each model's context window (up to 1M tokens in this work) to ensure a fair comparison. EMB-S pairs queries with collision-tested near-miss hard negatives and gold evidence sets spanning one or more documents, validated via human screening and LLM verification. We also propose a decoupled diagnostic protocol that reports evidence access (document-ID localization) separately from end-to-end QA quality under full-context prompting. This enables consistent diagnosis for both native long-context prompting and retrieval pipelines. Across a reference-corpus ladder from domain-isolated 64K contexts to a globally shared 326M-token environment, we observe a clear reality gap. Systems that saturate benign NIAH degrade sharply in evidence access under semantic interference. These results indicate that semantic discrimination, not context length alone, is the dominant bottleneck for long-context memory at scale.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 19 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

13 of 19 references.

References · 13

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  5. 2022Data Contamination: From Memorization to Exploitationcited by 236
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  8. 2025Survey on Evaluation of LLM-based Agentscited by 204
  9. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  10. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  11. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  12. 2024Long Context vs. RAG for LLMs: An Evaluation and Revisitscited by 26
1 more
  1. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matching
  2. 2026UNREAL: Unifying Retrieval and Long-Context with a Single Model
  3. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
  4. 2026MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG
  5. 2026LENS: In-Context Search via Latent Evidence Exploration over Dynamic Raw Documents
  6. 2026SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature
  7. 2026Relevance Is Not Sufficiency: What Actually Closes the Evidence Gap in Long-Term Memory QA
  8. 2026Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks