mlsanityData to 13 Oct 2026

Meta Knowledge Distillation

arXiv 2202.07940 ↗First posted 16 Feb 2022cs.LG cs.CVcited by 20

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Recent studies pointed out that knowledge distillation (KD) suffers from two degradation problems, the teacher-student gap and the incompatibility with strong data augmentations, making it not applicable to training state-of-the-art models, which are trained with advanced augmentations. However, we observe that a key factor, i.e., the temperatures in the softmax functions for generating probabilities of both the teacher and student models, was mostly overlooked in previous methods. With properly tuned temperatures, such degradation problems of KD can be much mitigated. However, instead of relying on a naive grid search, which shows poor transferability, we propose Meta Knowledge Distillation (MKD) to meta-learn the distillation with learnable meta temperature parameters. The meta parameters are adaptively adjusted during training according to the gradients of the learning objective. We validate that MKD is robust to different dataset scales, different teacher/student architectures, and different types of data augmentation. With MKD, we achieve the best performance with popular ViT architectures among compared methods that use only ImageNet-1K as training data, ranging from tiny to large models. With ViT-L, we achieve 86.5% with 600 epochs of training, 0.6% better than MAE that trains for 1,650 epochs.

Key facts. Cited by 20 papers in the map (data through 13 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 35 of its 43 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

35 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Isotonic Data Augmentation for Knowledge Distillation3 Jul
  2. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation3 Jul
  3. 2021Knowledge distillation: A good teacher is patient and consistent9 Jun
  4. 2021Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study1 Apr

Other references · 31

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  6. 2016Layer Normalizationcited by 5,624
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,266
19 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,631
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,067
  4. 2019When Does Label Smoothing Help?cited by 986
  5. 2016Deep Networks with Stochastic Depthcited by 977
  6. 2019Relational Knowledge Distillationcited by 881
  7. 2021Going deeper with Image Transformerscited by 762
  8. 2019Contrastive Representation Distillationcited by 739
  9. 2019Improved Knowledge Distillation via Teacher Assistantcited by 569
  10. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 566
  11. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 378
  12. 2019On the Efficacy of Knowledge Distillationcited by 378
  13. 2020Meta Pseudo Labelscited by 374
  14. 2019Variational Information Distillation for Knowledge Transfercited by 323
  15. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  16. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  17. 2018Meta-Gradient Reinforcement Learningcited by 158
  18. 2020Meta-Learning with Adaptive Hyperparameterscited by 79
  19. 2020An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillationcited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 5; 2025: 3; 2026: 1.

20 papers.

Sort:

By area:

  1. 2022Curriculum Temperature for Knowledge Distillationcited by 111
  2. 2024Logit Standardization in Knowledge Distillationcited by 102
  3. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  4. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  5. 2023Generic-to-Specific Distillation of Masked Autoencoderscited by 18
  6. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  7. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
  8. 2023Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcementcited by 11
  9. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11
  10. 2024To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DROcited by 11
  11. 2024Dynamic Temperature Knowledge Distillationcited by 6
  12. 2025Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillationcited by 6
8 more
  1. 2023Efficient Temporal Sentence Grounding in Videos with Multi-Teacher Knowledge Distillationcited by 3
  2. 2024Instance Temperature Knowledge Distillationcited by 3
  3. 2025Adaptive Temperature Based on Logits Correlation in Knowledge Distillationcited by 2
  4. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Frameworkcited by 1
  5. 2023Multilingual Sentence-Level Semantic Search using Meta-Distillation Learningcited by 1
  6. 2025Dynamic Temperature Scheduler for Knowledge Distillationcited by 1
  7. 2026Consistently Informative Soft-Label Temperature for Knowledge Distillationcited by 1
  8. 2022Hint-dynamic Knowledge Distillationcited by 0