mlsanityData to 14 Oct 2026

NeedleChain: Measuring Intact Context Comprehension Capability of Large Language Models

arXiv 2507.22411 ↗First posted 30 Jul 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent reports suggest that LLMs can handle increasingly long contexts. However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information. Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs. In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens. To evaluate this capability more rigorously, we introduce NeedleChain, a benchmark designed to test whether models can faithfully incorporate all given evidence. NeedleChain includes three variants that differ in the required order of comprehension, along with a parallel benchmark based on the needle-in-a-haystack(NIAH) paradigm. By comparing these variants, NeedleChain enables a more comprehensive assessment of context understanding. We further propose a training-free strategy that encourages models to reflect all available information, ROPE contraction, highlighting the importance of full-context integration and pointing to new directions for improving reliable reasoning over context.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

22 of 35 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  11. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  12. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
10 more
  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  2. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  3. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  4. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  5. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  6. 2024First-Person Fairness in Chatbotscited by 20
  7. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 15
  8. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  9. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
  10. 2024GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Modelscited by 5

Built on by

Not cited yet.