mlsanityData to 7 Oct 2026

Teaching Small Language Models to Reason

arXiv 2212.08410 ↗First posted 16 Dec 2022cs.CL cs.LGcited by 279

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain of thought prompting successfully improves the reasoning capabilities of large language models, achieving state of the art results on a range of datasets. However, these reasoning capabilities only appear to emerge in models with a size of over 100 billion parameters. In this paper, we explore the transfer of such reasoning capabilities to models with less than 100 billion parameters via knowledge distillation. Specifically, we finetune a student model on the chain of thought outputs generated by a larger teacher model. Our experiments show that the proposed method improves task performance across arithmetic, commonsense and symbolic reasoning datasets. For example, the accuracy of T5 XXL on GSM8K improves from 8.11% to 21.99% when finetuned on PaLM-540B generated chains of thought.

Key facts. Cited by 279 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 32 references that are in the map. First posted on arXiv 16 Dec 2022.

Builds on

23 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
9 more
  1. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  4. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  5. 2022UL2: Unifying Language Learning Paradigmscited by 292
  6. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  7. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  8. 2018Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graphcited by 67
  9. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 72; 2024: 72; 2025: 66; 2026: 62.

279 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  7. 2022A Survey on In-context Learningcited by 717
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  12. 2022Large Language Models Are Reasoning Teacherscited by 318
38 more
  1. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  3. 2023A Survey on Model Compression for Large Language Modelscited by 263
  4. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  5. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  9. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  12. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  13. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  14. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  15. 2025Thinkless: LLM Learns When to Thinkcited by 107
  16. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
  17. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  18. 2023Amortizing intractable inference in large language modelscited by 94
  19. 2025Efficient Reasoning Models: A Surveycited by 93
  20. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  21. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  22. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  23. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  24. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  25. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 64
  26. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  27. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  28. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  29. 2025Are Reasoning Models More Prone to Hallucination?cited by 62
  30. 2023Natural Language Reasoning, A Surveycited by 61
  31. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61
  32. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  33. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  34. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  35. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  36. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  37. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  38. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45

50 most cited of 279 All 279 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasoners
  2. 2022Distilling Reasoning Capabilities into Smaller Language Models
  3. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
  4. 2024General Purpose Verification for Chain of Thought Prompting
  5. 2023Implicit Chain of Thought Reasoning via Knowledge Distillation
  6. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  7. 2025Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction
  8. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information