mlsanityData to 7 Oct 2026

Distilling the Knowledge in a Neural Network

arXiv 1503.02531 ↗First posted 9 Mar 2015stat.ML cs.LG cs.NEcited by 10,076

Not on the map. Why?

Cited by 382 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.

Key facts. Cited by 10,076 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Mar 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 1; 2019: 4; 2020: 2; 2021: 1506; 2022: 1544; 2023: 1742; 2024: 1765; 2025: 1648; 2026: 1863.

10,076 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2024Depth Anything V2cited by 1,385
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  8. 2021Transformers in Vision: A Surveycited by 1,009
  9. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  10. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  13. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  14. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  15. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  16. 2021Robust fine-tuning of zero-shot modelscited by 785
  17. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
  18. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  19. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  20. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  21. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  22. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  23. 2022Large Language Models Can Self-Improvecited by 656
  24. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  25. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  26. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  27. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  28. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  29. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  30. 2024Large Language Models: A Surveycited by 506
  31. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  32. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  33. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  34. 2021Towards Personalized Federated Learningcited by 462
  35. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  36. 2021Intriguing Properties of Vision Transformerscited by 446
  37. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  38. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445

50 most cited of 10,076 All 10,076 (JSON)