mlsanityData to 7 Oct 2026

Knowledge distillation: A good teacher is patient and consistent

arXiv 2106.05237 ↗First posted 9 Jun 2021cs.CV cs.AI cs.LGcited by 255

Not on the map. Why?

Abstract (arXiv)

There is a growing discrepancy in computer vision between large-scale models that achieve state-of-the-art performance and models that are affordable in practical applications. In this paper we address this issue and significantly bridge the gap between these two types of models. Throughout our empirical investigation we do not aim to necessarily propose a new method, but strive to identify a robust and effective recipe for making state-of-the-art large scale models affordable in practice. We demonstrate that, when performed correctly, knowledge distillation can be a powerful tool for reducing the size of large models without compromising their performance. In particular, we uncover that there are certain implicit design choices, which may drastically affect the effectiveness of distillation. Our key contribution is the explicit identification of these design choices, which were not previously articulated in the literature. We back up our findings by a comprehensive empirical study, demonstrate compelling results on a wide range of vision datasets and, in particular, obtain a state-of-the-art ResNet-50 model for ImageNet, which achieves 82.8% top-1 accuracy.

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
29 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2019Searching for MobileNetV3cited by 1,984
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  7. 2018Group Normalizationcited by 1,514
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  11. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  12. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  13. 2020What is the State of Neural Network Pruning?cited by 578
  14. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  15. 2019Selective Kernel Networkscited by 401
  16. 2019On the Efficacy of Knowledge Distillationcited by 377
  17. 2018Rethinking ImageNet Pre-trainingcited by 376
  18. 2020Rethinking Pre-training and Self-trainingcited by 352
  19. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  20. 2019Fixing the train-test resolution discrepancycited by 209
  21. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  22. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  23. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  24. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  25. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  26. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  27. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  28. 2020MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Trickscited by 39
  29. 2017Combining Weakly and Webly Supervised Learning for Classifying Food Imagescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 39; 2023: 79; 2024: 59; 2025: 43; 2026: 27.

255 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  3. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  4. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  5. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  8. 2021Does Knowledge Distillation Really Work?cited by 187
  9. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  10. 2022FlexiViT: One Model for All Patch Sizescited by 151
  11. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  12. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
38 more
  1. 2024MobileNetV4 -- Universal Models for the Mobile Ecosystemcited by 98
  2. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  3. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  4. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 63
  5. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  6. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  7. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  8. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  9. 2024Tiny Machine Learning: Progress and Futurescited by 40
  10. 2023Initializing Models with Larger Onescited by 39
  11. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  12. 2023A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentationcited by 33
  13. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  14. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  15. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  16. 2022TeST: Test-time Self-Training under Distribution Shiftcited by 29
  17. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  18. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  19. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  20. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  21. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 26
  22. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  23. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 25
  24. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  25. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  26. 2024iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrievalcited by 24
  27. 2021Network Augmentation for Tiny Deep Learningcited by 23
  28. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  29. 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillationcited by 21
  30. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 20
  31. 2022Meta Knowledge Distillationcited by 20
  32. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  33. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 20
  34. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  35. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20
  36. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  37. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  38. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19

50 most cited of 255 All 255 (JSON)