mlsanityData to 7 Oct 2026

Long-context Language Models Fail in Basic Retrieval Tasks Without Sufficient Reasoning Steps

arXiv 2410.04422 ↗First posted 6 Oct 2024cs.CLcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Long-context language models (LCLMs), characterized by their extensive context window, are becoming popular. However, despite the fact that they are nearly perfect at standard long-context retrieval tasks, our evaluations demonstrate they fail in some basic cases. Later, we find they can be well addressed with a sufficient number of reasoning steps, guided by specific CoT prompts. This result emphasizes the potential necessity of solving specific long-context tasks using long-CoT methods, while previous long-context benchmarks always ignore the necessity of long reasoning for long-context tasks and treat them as direct QA tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 17 of its 25 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

17 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities16 Jul

Other references · 16

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  11. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  12. 2023Retrieval meets Long Context Large Language Modelscited by 114
4 more
  1. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  2. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  3. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
  4. 2024Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understandingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Taskscited by 0
  2. 2024Patience Is The Key to Large Language Model Reasoningcited by 0
  3. 2026Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
  2. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?
  3. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering
  4. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
  5. 2024Evaluating Multilingual Long-Context Models for Retrieval and Reasoning
  6. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
  7. 2025Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
  8. 2025Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding