mlsanityData to 14 Oct 2026

Towards a theory of model distillation

arXiv 2403.09053 ↗First posted 14 Mar 2024cs.LG cs.AI cs.NEcited by 15

Not on the map. Why?

Abstract (arXiv)

Distillation is the task of replacing a complicated machine learning model with a simpler model that approximates the original [BCNM06,HVD15]. Despite many practical applications, basic questions about the extent to which models can be distilled, and the runtime and amount of data needed to distill, remain largely open. To study these questions, we initiate a general theory of distillation, defining PAC-distillation in an analogous way to PAC-learning [Val84]. As applications of this theory: (1) we propose new algorithms to extract the knowledge stored in the trained weights of neural networks -- we show how to efficiently distill neural networks into succinct, explicit decision tree representations when possible by using the ``linear representation hypothesis''; and (2) we prove that distillation can be much cheaper than learning from scratch, and make progress on characterizing its complexity.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 67 of its 110 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

67 of 110 references.

References · 67

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  8. 2022Toy Models of Superpositioncited by 971
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
38 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  2. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  3. 2020What is the State of Neural Network Pruning?cited by 583
  4. 2014Convex Optimization: Algorithms and Complexitycited by 575
  5. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  6. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  7. 2015Deep Learning with Limited Numerical Precisioncited by 376
  8. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  9. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  10. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  11. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  12. 2018Model compression via distillation and quantizationcited by 289
  13. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  14. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  15. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  16. 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
  17. 2019Structured Pruning of Large Language Modelscited by 203
  18. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  19. 2021Thinking Like Transformerscited by 184
  20. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  21. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  22. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  23. 2021Towards Understanding Knowledge Distillationcited by 158
  24. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  25. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  26. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  27. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  28. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  29. 2022Linear Adversarial Concept Erasurecited by 101
  30. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  31. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  32. 2017Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillationcited by 62
  33. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  34. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  35. 2023Transformers learn through gradual rank increasecited by 52
  36. 2019Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Modelscited by 42
  37. 2017Interpretability via Model Extractioncited by 40
  38. 2021Adaptive wavelet distillation from neural networks through interpretationscited by 32

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledgecited by 10
  2. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  3. 2024A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentialitycited by 8
  4. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  5. 2025Expand Neurons, Not Parameterscited by 3
  6. 2025Secret mixtures of experts inside your LLMcited by 2
  7. 2024Optimizing Vision Transformers with Data-Free Knowledge Transfercited by 1
  8. 2026Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Modelscited by 1
  9. 2026Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Modelscited by 1
  10. 2026Learning through Internalizationcited by 1
  11. 2024Fast decision tree learning solves hard coding-theoretic problemscited by 0
  12. 2024Finding the Underlying Viscoelastic Constitutive Equation via Universal Differential Equations and Differentiable Physicscited by 0
3 more
  1. 2025Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Methodcited by 0
  2. 2026Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competitioncited by 0
  3. 2026A Survey on the Linear Representation Hypothesiscited by 0