mlsanityData to 14 Oct 2026

Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision

arXiv 2502.20790 ↗First posted 28 Feb 2025cs.CLcited by 21

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have highlighted the challenge of handling long-context tasks, where models need to reason over extensive input contexts to aggregate target information. While Chain-of-Thought (CoT) prompting has shown promise for multi-step reasoning, its effectiveness for long-context scenarios remains underexplored. Through systematic investigation across diverse tasks, we demonstrate that CoT's benefits generalize across most long-context scenarios and amplify with increasing context length. Motivated by this critical observation, we propose LongRePS, a process-supervised framework that teaches models to generate high-quality reasoning paths for enhanced long-context performance. Our framework incorporates a self-sampling mechanism to bootstrap reasoning paths and a novel quality assessment protocol specifically designed for long-context scenarios. Experimental results on various long-context benchmarks demonstrate the effectiveness of our approach, achieving significant improvements over outcome supervision baselines on both in-domain tasks (+13.6/+3.8 points for LLaMA/Qwen on MuSiQue) and cross-domain generalization (+9.3/+8.1 points on average across diverse QA tasks). Our code, data and trained models are made public to facilitate future research.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 50 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

33 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning18 Sep
  2. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
19 more
  1. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  4. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  6. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  8. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  9. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  10. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  11. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  12. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  13. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  14. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  15. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 50
  16. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  17. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  18. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoningcited by 26
  19. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 8.

21 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  3. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  4. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  5. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewardscited by 25
  6. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  7. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Draftingcited by 15
  8. 2025LastingBench: Defend Benchmarks Against Knowledge Leakagecited by 14
  9. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  10. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  11. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  12. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimizationcited by 9
9 more
  1. 2026Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoningcited by 7
  2. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  3. 2026Read More, Think More: Revisiting Observation Reduction for Web Agentscited by 4
  4. 2025Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimizationcited by 3
  5. 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systemscited by 2
  6. 2025Structured Reasoning with Tree-of-Thoughts for Bengali Math Word Problemscited by 1
  7. 2026Probing How Scalable Table Data Enhances General Long-Context Reasoningcited by 1
  8. 2026Can LLM Teams Play What? Where? When?cited by 0
  9. 2026Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  2. 2025Long Chain-of-Thought Reasoning Across Languages
  3. 2025Fractured Chain-of-Thought Reasoning
  4. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  5. 2024Chain-of-Thought Augmentation with Logit Contrast for Enhanced Reasoning in Language Models
  6. 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval
  7. 2023Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models
  8. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting