mlsanityData to 14 Oct 2026

Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation

arXiv 2610.17607 ↗First posted 13 Oct 2026cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

While long chain-of-thought (CoT) distillation significantly improves the reasoning abilities of language models, its underlying mechanism remains poorly understood. Current pipelines generally rely on rejection sampling to gather reasoning traces across diverse prompts, assuming that factual correctness and broad problem coverage are indispensable. Surprisingly, we find that training exclusively on flawed reasoning trajectories or a restricted prompt set yields nearly comparable performance. Driven by this counterintuitive finding, we propose the Behavioral Hypothesis: the essence of Long-CoT distillation is behavioral alignment with structured reasoning patterns, rather than factual knowledge memorization. We then validate this by demonstrating substantial reasoning gains even when models are fine-tuned exclusively on previously mastered problems-effectively isolating behavioral alignment from novel knowledge injection. Having established that reasoning patterns are the true bottleneck, we investigate how to learn them across varying levels of complexity optimally. We reveal a critical interplay: while the recent probability-based loss excels at learning simple patterns, the standard cross-entropy loss is essential for scaling to complex CoTs. We believe these insights demystify CoT distillation and provide a principled foundation for the future development of reasoning models. Code is available on https://github.com/yuyongcan/Long-CoT-Distillation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 51 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

41 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks24 Dec
  2. 2025What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT23 Sep
  3. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillation14 Jul
  4. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation20 Mar

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
25 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  6. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  7. 2025Thyme: Think Beyond Imagescited by 167
  8. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  9. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  10. 2023Learning From Mistakes Makes LLM Better Reasonercited by 101
  11. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  12. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  13. 2023Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Datacited by 21
  14. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  15. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  16. 2025Proximal Supervised Fine-Tuningcited by 16
  17. 2025Anchored Supervised Fine-Tuningcited by 15
  18. 2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capabilitycited by 14
  19. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-contextcited by 13
  20. 2025DeepResearch-Slice: Bridging the Retrieval-Utilization Gap via Explicit Text Slicingcited by 9
  21. 2026SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuningcited by 7
  22. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 6
  23. 2025Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoningcited by 5
  24. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 5
  25. 2026Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  2. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
  3. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs
  4. 2025Lost at the Beginning of Reasoning
  5. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning
  6. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  7. 2025Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models
  8. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought