# NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities

arXiv: https://arxiv.org/abs/2407.11963 · HTML: https://mlsanity.com/paper/2407.11963/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Skywork: A More Open Bilingual Foundation Model](https://mlsanity.com/paper/2310.19341/) (2023)
- [Training-Free Long-Context Scaling of Large Language Models](https://mlsanity.com/paper/2402.17463/) (2024)
- [Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?](https://mlsanity.com/paper/2406.13121/) (2024)
- [R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason](https://mlsanity.com/paper/1910.04601/) (2019)

## Built on by (most cited first, 30 of 32)

- [Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities](https://mlsanity.com/paper/2511.02817/) (2025)
- [Reliable Weak-to-Strong Monitoring of LLM Agents](https://mlsanity.com/paper/2508.19461/) (2025)
- [Large Language Models Can Self-Improve in Long-context Reasoning](https://mlsanity.com/paper/2411.08147/) (2024)
- [MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2505.10610/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs](https://mlsanity.com/paper/2410.04698/) (2024)
- [FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression](https://mlsanity.com/paper/2507.20030/) (2025)
- [NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts](https://mlsanity.com/paper/2506.02000/) (2025)
- [PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts](https://mlsanity.com/paper/2508.09848/) (2025)
- [MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models](https://mlsanity.com/paper/2605.14906/) (2026)
- [When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression](https://mlsanity.com/paper/2605.08234/) (2026)
- [Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps](https://mlsanity.com/paper/2410.04422/) (2024)
- [Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding)](https://mlsanity.com/paper/2502.17169/) (2025)
- [Do Reasoning LLMs Refuse What They Infer in Long Contexts?](https://mlsanity.com/paper/2602.08874/) (2026)
- [Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG](https://mlsanity.com/paper/2412.06078/) (2024)
- [Can Memory-Augmented Language Models Generalize on Reasoning-in-a-Haystack Tasks?](https://mlsanity.com/paper/2503.07903/) (2025)
- [Reasoning on Multiple Needles In A Haystack](https://mlsanity.com/paper/2504.04150/) (2025)
- [Getting out of the Big-Muddy: Escalation of Commitment in LLMs](https://mlsanity.com/paper/2508.01545/) (2025)
- [Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding](https://mlsanity.com/paper/2603.10863/) (2026)
- [KV Cache Offloading for Context-Intensive Tasks](https://mlsanity.com/paper/2604.08426/) (2026)
- [From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs](https://mlsanity.com/paper/2606.09508/) (2026)
- [Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning](https://mlsanity.com/paper/2606.18831/) (2026)
- [Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models](https://mlsanity.com/paper/2607.19257/) (2026)
- [Context Is Not Comprehension](https://mlsanity.com/paper/2506.04907/) (2025)
- [Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs](https://mlsanity.com/paper/2601.02023/) (2026)
- [Status Hierarchies in Language Models](https://mlsanity.com/paper/2601.17577/) (2026)
- [Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries](https://mlsanity.com/paper/2604.06416/) (2026)
- [MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios](https://mlsanity.com/paper/2606.14747/) (2026)
- [WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning](https://mlsanity.com/paper/2607.09328/) (2026)
- [MGAL: A Multilingual Granularity-Aware Long-Context Benchmark](https://mlsanity.com/paper/2608.20853/) (2026)

## Abstract (arXiv)

The capability of large language models to handle long-context information is crucial across various real-world applications. Existing evaluation methods often rely either on real-world long texts, making it difficult to exclude the influence of models' inherent knowledge, or introduce irrelevant filler content to artificially achieve target lengths, reducing assessment effectiveness. To address these limitations, we introduce NeedleBench, a synthetic framework for assessing retrieval and reasoning performance in bilingual long-context tasks with adaptive context lengths. NeedleBench systematically embeds key data points at varying depths to rigorously test model capabilities. Tasks are categorized into two scenarios: information-sparse, featuring minimal relevant details within extensive irrelevant text to simulate simple retrieval tasks; and information-dense (the Ancestral Trace Challenge), where relevant information is continuously distributed throughout the context to simulate complex reasoning tasks. Our experiments reveal that although recent reasoning models like Deepseek-R1 and OpenAI's o3 excel in mathematical reasoning, they struggle with continuous retrieval and reasoning in information-dense scenarios, even at shorter context lengths. We also characterize a phenomenon termed 'under-thinking', where models prematurely conclude reasoning despite available information. NeedleBench thus provides critical insights and targeted tools essential for evaluating and improving LLMs' long-context capabilities. All resources are available at OpenCompass: https://github.com/open-compass/opencompass.