mlsanityData to 14 Oct 2026

Reasoning on Multiple Needles In A Haystack

arXiv 2504.04150 ↗First posted 5 Apr 2025cs.CL cs.AIcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The Needle In A Haystack (NIAH) task has been widely used to evaluate the long-context question-answering capabilities of Large Language Models (LLMs). However, its reliance on simple retrieval limits its effectiveness. To address this limitation, recent studies have introduced the Multiple Needles In A Haystack Reasoning (MNIAH-R) task, which incorporates supporting documents (Multiple needles) of multi-hop reasoning tasks into a distracting context (Haystack}). Despite this advancement, existing approaches still fail to address the issue of models providing direct answers from internal knowledge, and they do not explain or mitigate the decline in accuracy as context length increases. In this paper, we tackle the memory-based answering problem by filtering out direct-answer questions, and we reveal that performance degradation is primarily driven by the reduction in the length of the thinking process as the input length increases. Building on this insight, we decompose the thinking process into retrieval and reasoning stages and introduce a reflection mechanism for multi-round extension. We also train a model using the generated iterative thinking process, which helps mitigate the performance degradation. Furthermore, we demonstrate the application of this retrieval-reflection capability in mathematical reasoning scenarios, improving GPT-4o's performance on AIME2024.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 43 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

24 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries19 Sep
  2. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities16 Jul
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack14 Jun

Other references · 21

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  12. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
9 more
  1. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  2. 2024Many-Shot In-Context Learningcited by 206
  3. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  4. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  5. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  6. 2024Long Context Compression with Activation Beaconcited by 104
  7. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  8. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  9. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  2. 2026Characterize Then Distill: Mechanistic Reasoning in Large Output Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
  2. 2026Exploring the Meta-level Reasoning of Large Language Models via a Tool-based Multi-hop Tabular Question Answering Task
  3. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
  4. 2026Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
  6. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
  7. 2022Rethinking with Retrieval: Faithful Large Language Model Inference
  8. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find