Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models. However, the impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored. This gap in understanding is particularly significant given the increasing importance of Retrieval-Augmented Generation (RAG) systems, where efficient acquisition and utilization of contextual information are paramount for generating reliable responses. Motivated by the need to understand how the extended long-CoT process influences long-context comprehension, we conduct a comprehensive investigation using a series of open-source models distilled from Deepseek-R1, renowned for its exceptional reasoning capabilities. Our study focuses on evaluating these models' performance in extracting and integrating relevant information from extended contexts through multi-document question and answering tasks. Through rigorous experimentation, we demonstrate that distilled reasoning patterns significantly improve long-context understanding. Our analysis reveals that distillation fosters greater long-context awareness by promoting more detailed and explicit reasoning processes during context analysis and information parsing. This advancement effectively mitigates the persistent "lost in the middle" issue that has hindered long-context models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 34 of its 36 references that are in the map. First posted on arXiv 20 Jul 2025.
Builds on
34 of 36 references.
Closest ancestors · 3
Same direction, newest first
- 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models14 Jan
- 2023Code Llama: Open Foundation Models for Code24 Aug
- 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
Other references · 31
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
- 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
- 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
19 more
- 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
- 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
- 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
- 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
- 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
- 2024Make Your LLM Fully Utilize the Contextcited by 118
- 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
- 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
- 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
- 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
- 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
- 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
- 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
- 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
- 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
- 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
- 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
- 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
- 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
- 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
- 2024Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding
- 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval
- 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs
- 2026Targeted Retrieval, Compact Representations: How CoT Reasoning Improves Long-Context Counting
- 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
- 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion