mlsanityData to 7 Oct 2026

Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning

arXiv 2502.18001 ↗First posted 25 Feb 2025cs.CLcited by 40

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a non-monotonic relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has minimal effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do NOT always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs. The code and datasets are available at https://github.com/EIT-NLP/Distilling-CoT-Reasoning.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 38 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

28 of 38 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selection30 Nov
  2. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?31 Oct
  3. 2024Distilling System 2 into System 18 Jul
  4. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning1 Jul
  5. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thought28 May
  6. 2024The Impact of Reasoning Step Length on Large Language Models10 Jan
  7. 2022Large Language Models Are Reasoning Teachers20 Dec
  8. 2022Teaching Small Language Models to Reason16 Dec
  9. 2022Distilling Reasoning Capabilities into Smaller Language Models1 Dec
  10. 2022Large Language Models are Zero-Shot Reasoners24 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  12. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
6 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  3. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  4. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  5. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  6. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 22.

40 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Efficient Reasoning Models: A Surveycited by 93
  3. 2025Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Timecited by 48
  4. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  5. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Thinkcited by 23
  6. 2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
  7. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotationscited by 14
  8. 2025Latent Chain-of-Thought World Modeling for End-to-End Drivingcited by 14
  9. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  10. 2025Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoningcited by 11
  11. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 10
  12. 2025Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?cited by 9
28 more
  1. 2025Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoningcited by 8
  2. 2025The Signal is in the Steps: Local Scoring for Reasoning Data Selectioncited by 7
  3. 2025Merge-of-Thought Distillationcited by 6
  4. 2025Harnessing Chain-of-Thought Metadata for Task Routing and Adversarial Prompt Detectioncited by 5
  5. 2026The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoningcited by 4
  6. 2025Policy Frameworks for Transparent Chain-of-Thought Reasoning in Large Language Modelscited by 3
  7. 2026Select to Think: Unlocking SLM Potential with Local Sufficiencycited by 3
  8. 2026Self-Policy Distillation via Capability-Selective Subspace Projectioncited by 3
  9. 2026Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Modelscited by 2
  10. 2025Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoningcited by 1
  11. 2026"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Frameworkcited by 1
  12. 2026Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversioncited by 1
  13. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  14. 2026When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviorscited by 1
  15. 2025Systematic Optimization of Open Source Large Language Models for Mathematical Reasoningcited by 0
  16. 2025Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Seriescited by 0
  17. 2025Conscious Data Contribution via Community-Driven Chain-of-Thought Distillationcited by 0
  18. 2026Emergent Manifold Separability during Reasoning in Large Language Modelscited by 0
  19. 2026Metacognitive Behavioral Tuning of Large Language Models for Multi-Hop Question Answeringcited by 0
  20. 2026Structural Rationale Distillation via Reasoning Space Compressioncited by 0
  21. 2026Distribution Corrected Offline Data Distillation for Large Language Modelscited by 0
  22. 2026Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibilitycited by 0
  23. 2026SchGen: PCB Schematic Generation with Semantic-Grounded Code Representationscited by 0
  24. 2026Characterize Then Distill: Mechanistic Reasoning in Large Output Spacescited by 0
  25. 2026Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichmentcited by 0
  26. 2026AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agentscited by 0
  27. 2026Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understandingcited by 0
  28. 2026Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
  2. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  3. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
  4. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking
  5. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
  6. 2025The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
  7. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales
  8. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters