mlsanityData to 7 Oct 2026

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

arXiv 2305.02301 ↗First posted 3 May 2023cs.CL cs.AI cs.LGcited by 710

Not on the map. Why?

Cited by 160 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to few-shot prompted LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80% of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100% of the dataset. We release the code at: https://github.com/google-research/distilling-step-by-step .

Key facts. Cited by 710 papers in the map (data through 7 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

47 of 58 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
35 more
  1. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  8. 2022Large Language Models Can Self-Improvecited by 656
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  11. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  12. 2022Large Language Models Are Reasoning Teacherscited by 318
  13. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  14. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
  15. 2022Teaching Small Language Models to Reasoncited by 279
  16. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  17. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  18. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  19. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  20. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  21. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  22. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  23. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  24. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  25. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  26. 2018Model Reconstruction from Model Explanationscited by 84
  27. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  28. 2018Knowledge Transfer with Jacobian Matchingcited by 73
  29. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  30. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  31. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  32. 2016Rationale-Augmented Convolutional Neural Networks for Text Classificationcited by 39
  33. 2022QAmeleon: Multilingual QA with Only 5 Examplescited by 25
  34. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  35. 2022Weighted Distillation with Unlabeled Examplescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 89; 2024: 224; 2025: 174; 2026: 223.

710 papers.

Sort:

By area:

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  3. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  4. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  5. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  6. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  8. 2023A Survey on Model Compression for Large Language Modelscited by 263
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  10. 2023The False Promise of Imitating Proprietary LLMscited by 254
  11. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
38 more
  1. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  2. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  5. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  6. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  7. 2024The pitfalls of next-token predictioncited by 156
  8. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  9. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  10. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  11. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  12. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  13. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  14. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  15. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  16. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  17. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  18. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  19. 2024Knowledge Fusion of Large Language Modelscited by 110
  20. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrappingcited by 96
  21. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  22. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  23. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  24. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  25. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  26. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  27. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  28. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  29. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  30. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  31. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  32. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  33. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  34. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 70
  35. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  36. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  37. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  38. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 68

50 most cited of 710 All 710 (JSON)