mlsanityData to 14 Oct 2026

Distillation Scaling Laws

arXiv 2502.08606 ↗First posted 12 Feb 2025cs.LG cs.AI cs.CLcited by 67

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 65 of its 127 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

65 of 127 references.

References · 65

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2023Mistral 7Bcited by 3,164
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  9. 2024DeepSeek-V3 Technical Reportcited by 811
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024Phi-4 Technical Reportcited by 711
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  13. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  14. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  15. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  16. 2023Scaling Data-Constrained Language Modelscited by 403
  17. 2021Explaining Neural Scaling Lawscited by 363
  18. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  19. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  20. 2021Scaling Laws for Transfercited by 272
  21. 2015Unifying distillation and privileged informationcited by 233
  22. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  23. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  24. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  25. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  26. 2023A Spectral Condition for Feature Learningcited by 109
  27. 2021The Efficiency Misnomercited by 98
  28. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  29. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  30. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 80
  31. 2023On the Calibration of Large Language Models and Alignmentcited by 73
  32. 2024Chinchilla Scaling: A replication attemptcited by 72
  33. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  34. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 50
  35. 2019SciPy 1.0--Fundamental Algorithms for Scientific Computing in Pythoncited by 49
  36. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  37. 2024The Elements of Differentiable Programmingcited by 43
  38. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 37.

67 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  3. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  4. 2026Ministral 3cited by 114
  5. 2025Scaling Laws for Optimal Data Mixturescited by 52
  6. 2025Scaling Laws of Synthetic Data for Language Modelscited by 46
  7. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  8. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 27
  9. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 25
  10. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  11. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  12. 2026DOPD: Dual On-policy Distillationcited by 12
38 more
  1. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 11
  2. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 10
  3. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  4. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationalescited by 8
  5. 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignmentcited by 7
  6. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  7. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  8. 2025Synthetic bootstrapped pretrainingcited by 6
  9. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  10. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  11. 2025Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theorycited by 5
  12. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 5
  13. 2025Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuningcited by 4
  14. 2025Scalable Strategies for Continual Learning with Replaycited by 4
  15. 2025Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deploymentcited by 4
  16. 2026Self-Distillation for Multi-Token Predictioncited by 4
  17. 2025Scaling Laws for Data-Efficient Visual Transfer Learningcited by 3
  18. 2025Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensemblescited by 3
  19. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  20. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 3
  21. 2026Strong Teacher Not Needed? On Distillation in LLM Pretrainingcited by 3
  22. 2026Skaling: Chinchilla's Exponents Meet Kaplan's Couplingcited by 3
  23. 2026LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristicscited by 3
  24. 2025Pruning Laws for Large Language Modelscited by 2
  25. 2025HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modelingcited by 2
  26. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  27. 2025Beyond Real Data: Synthetic Data through the Lens of Regularizationcited by 2
  28. 2026EUGens: Efficient, Unified, and General Dense Layerscited by 2
  29. 2026Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Modelscited by 2
  30. 2026Scaling Properties of Continuous Diffusion Spoken Language Modelscited by 1
  31. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  32. 2026Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Updatecited by 1
  33. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Modelscited by 0
  34. 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQAcited by 0
  35. 2025Towards Characterizing Knowledge Distillation of PPG Heart Rate Estimation Modelscited by 0
  36. 2026Metacognitive Behavioral Tuning of Large Language Models for Multi-Hop Question Answeringcited by 0
  37. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Modelscited by 0
  38. 2026Locking Pretrained Weights via Deep Low-Rank Residual Distillationcited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DisWOT: Student Architecture Search for Distillation WithOut Training
  2. 2026IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
  3. 2026D3-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
  4. 2022Revisiting Self-Distillation
  5. 2026SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
  6. 2021Does Knowledge Distillation Really Work?
  7. 2026Reference-Based Distillation Detection in LLMs
  8. 2022Robust Distillation for Worst-class Performance