mlsanityData to 7 Oct 2026

Large Language Models Are Reasoning Teachers

arXiv 2212.10071 ↗First posted 20 Dec 2022cs.CL cs.AI cs.LGcited by 318

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune-CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model's ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models. Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 75 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

50 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Teaching Small Language Models to Reason16 Dec
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
34 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2019When Does Label Smoothing Help?cited by 985
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  13. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  14. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  15. 2016Sequence-Level Knowledge Distillationcited by 761
  16. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  17. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  18. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  19. 2022Large Language Models Can Self-Improvecited by 656
  20. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  21. 2016Solving General Arithmetic Word Problemscited by 353
  22. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  23. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  24. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  25. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  26. 2022Language models show human-like content effects on reasoning taskscited by 171
  27. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  28. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  29. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  30. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  31. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 119
  32. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  33. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  34. 2020Progressive Network Grafting for Few-Shot Knowledge Distillationcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 79; 2024: 104; 2025: 74; 2026: 61.

318 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  5. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  8. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023Augmented Language Models: a Surveycited by 398
  12. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
38 more
  1. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  2. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023A Survey on Model Compression for Large Language Modelscited by 263
  5. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  6. 2024Token-Budget-Aware LLM Reasoningcited by 242
  7. 2023Large Language Model Alignment: A Surveycited by 241
  8. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  9. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  10. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  11. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  12. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  13. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
  14. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  15. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  16. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  17. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  18. 2024Large Language Models Meet NLP: A Surveycited by 102
  19. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  20. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  21. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  22. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  23. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  24. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  25. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  26. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 64
  27. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  28. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  29. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  30. 2023Natural Language Reasoning, A Surveycited by 61
  31. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  32. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  33. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  34. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  35. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  36. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  37. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoningcited by 46
  38. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46

50 most cited of 318 All 318 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
  2. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
  3. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  4. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  5. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  6. 2024The Impact of Reasoning Step Length on Large Language Models
  7. 2025Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
  8. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking