mlsanityData to 14 Oct 2026

Distilling Reasoning Capabilities into Smaller Language Models

arXiv 2212.00193 ↗First posted 1 Dec 2022cs.LG cs.CLcited by 158

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Step-by-step reasoning approaches like chain of thought (CoT) have proved to be very effective in inducing reasoning capabilities in large language models. However, the success of the CoT approach is fundamentally tied to the model size, and billion parameter-scale models are often needed to get CoT to work. In this paper, we propose a knowledge distillation approach that leverages the step-by-step CoT reasoning capabilities of larger models and distills these abilities into smaller models. In this work, we propose an alternative reasoning scheme, Socratic CoT, that learns a decomposition of the original problem into a sequence of subproblems and uses it to guide the intermediate reasoning steps. We use Socratic CoT to train a combination of two small distilled models: a problem decomposer and a subproblem solver. In practice, given a new problem, the two distilled models work in sync to decompose and solve complex problems. On multiple reasoning datasets (GSM8K, StrategyQA, and SVAMP), our proposed distillation strategies boosts the performance of smaller models over 70% compared to the baselines. Finally, we investigate when Socratic CoT is an effective alternative to CoT, demonstrating cases where a much smaller model (GPT-2 large) can outperform a 10X larger model (GPT-3 6B). Our code is available here: https://github.com/kumar-shridhar/Distiiling-LM

Key facts. Cited by 158 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 42 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

33 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
17 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  7. 2013Do Deep Nets Really Need to be Deep?cited by 474
  8. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  9. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  10. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  11. 2022Learning by Distilling Contextcited by 138
  12. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  13. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  14. 2019Answer-based Adversarial Training for Generating Clarification Questionscited by 43
  15. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 43
  16. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  17. 2019Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worldscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 22; 2024: 59; 2025: 47; 2026: 29.

158 papers.

Sort:

By area:

  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  2. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  3. 2023A Survey on Model Compression for Large Language Modelscited by 269
  4. 2023Challenges and Applications of Large Language Modelscited by 254
  5. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 226
  6. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  7. 2024The pitfalls of next-token predictioncited by 160
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  9. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  10. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 110
  11. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 101
  12. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
38 more
  1. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  3. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  4. 2024A Survey on Transformer Compressioncited by 46
  5. 2024HiGPT: Heterogeneous Graph Language Modelcited by 45
  6. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 43
  7. 2024Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Tracescited by 43
  8. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  9. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  10. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoningcited by 40
  11. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  12. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  13. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  14. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 30
  15. 2023The ART of LLM Refinement: Ask, Refine, and Trustcited by 29
  16. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  17. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  18. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 25
  19. 2025Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillationcited by 25
  20. 2025Antidistillation Samplingcited by 25
  21. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  22. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  23. 2024Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Miningcited by 20
  24. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  25. 2024Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfercited by 19
  26. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoningcited by 19
  27. 2024Contextualization Distillation from Large Language Model for Knowledge Graph Completioncited by 18
  28. 2023SCREWS: A Modular Framework for Reasoning with Revisionscited by 16
  29. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 16
  30. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  31. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillationcited by 14
  32. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14
  33. 2023Democratizing Reasoning Ability: Tailored Learning from Large Language Modelcited by 13
  34. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  35. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  36. 2023Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLMcited by 12
  37. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 12
  38. 2024Distilling Text Style Transfer With Self-Explanation From LLMscited by 12

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
  2. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  3. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales
  4. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  5. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  6. 2025Efficient Long CoT Reasoning in Small Language Models
  7. 2025Small Models Struggle to Learn from Strong Reasoners
  8. 2023Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA