mlsanityData to 7 Oct 2026

Adaptive Multi-Teacher Multi-level Knowledge Distillation

arXiv 2103.04062 ↗First posted 6 Mar 2021cs.CVcited by 69

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation~(KD) is an effective learning paradigm for improving the performance of lightweight student networks by utilizing additional supervision knowledge distilled from teacher networks. Most pioneering studies either learn from only a single teacher in their distillation learning methods, neglecting the potential that a student can learn from multiple teachers simultaneously, or simply treat each teacher to be equally important, unable to reveal the different importance of teachers for specific examples. To bridge this gap, we propose a novel adaptive multi-teacher multi-level knowledge distillation learning framework~(AMTML-KD), which consists two novel insights: (i) associating each teacher with a latent representation to adaptively learn instance-level teacher importance weights which are leveraged for acquiring integrated soft-targets~(high-level knowledge) and (ii) enabling the intermediate-level hints~(intermediate-level knowledge) to be gathered from multiple teachers by the proposed multi-group hint strategy. As such, a student model can learn multi-level knowledge from multiple teachers through AMTML-KD. Extensive results on publicly available datasets demonstrate the proposed learning framework ensures student to achieve improved performance than strong competitors.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

17 of 31 references.

References · 17

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2019Relational Knowledge Distillationcited by 873
  7. 2017Deep Mutual Learningcited by 621
  8. 2013Do Deep Nets Really Need to be Deep?cited by 466
  9. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  10. 2015Unifying distillation and privileged informationcited by 218
  11. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  12. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 113
5 more
  1. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  2. 2016Deep Model Compression: Distilling Knowledge from Noisy Teacherscited by 35
  3. 2019Unifying Heterogeneous Classifiers with Distillationcited by 24
  4. 2017Knowledge Adaptation: Teaching to Adaptcited by 17
  5. 2018Neural Compatibility Modeling with Attentive Knowledge Distillationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 11; 2024: 16; 2025: 12; 2026: 14.

69 papers.

Sort:

By area:

  1. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  2. 2021Zero-shot Adversarial Quantizationcited by 44
  3. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 39
  4. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  5. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  6. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  7. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  8. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 24
  9. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  10. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22
  11. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 22
  12. 2022NICEST: Noisy Label Correction and Training for Robust Scene Graph Generationcited by 19
38 more
  1. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  2. 2024PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillationcited by 17
  3. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  4. 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillationcited by 14
  5. 2023Distillation from Heterogeneous Models for Top-K Recommendationcited by 13
  6. 2023Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillationcited by 13
  7. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  8. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  9. 2025A Comprehensive Survey on Imbalanced Data Learningcited by 10
  10. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillationcited by 9
  11. 2026Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMscited by 9
  12. 2023Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertaintycited by 7
  13. 2024NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Modelscited by 7
  14. 2023Partial Network Cloningcited by 6
  15. 2022Adaptive Mixing of Auxiliary Losses in Supervised Learningcited by 5
  16. 2022Continual Knowledge Distillation for Neural Machine Translationcited by 4
  17. 2023Distilling from Similar Tasks for Transfer Learning on a Budgetcited by 4
  18. 2024The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillationcited by 4
  19. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  20. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  21. 2022Label driven Knowledge Distillation for Federated Learning with non-IID Datacited by 3
  22. 2023Few Shot Rationale Generation using Self-Training with Dual Teacherscited by 3
  23. 2024Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillationcited by 3
  24. 2025Mitigating Negative Flips via Margin Preserving Trainingcited by 3
  25. 2026Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregationcited by 3
  26. 2022PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensorscited by 2
  27. 2024Pairwise Distance Distillation for Unsupervised Real-World Image Super-Resolutioncited by 2
  28. 2025Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizercited by 2
  29. 2026Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMscited by 2
  30. 2023AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inferencecited by 1
  31. 2023DMT: Comprehensive Distillation with Multiple Self-supervised Teacherscited by 1
  32. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Expertscited by 1
  33. 2024GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillationcited by 1
  34. 2025Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Datacited by 1
  35. 2026Adaptive Weighting in Knowledge Distillation: An Axiomatic Framework for Multi-Scale Teacher Ensemble Optimizationcited by 1
  36. 2026Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priorscited by 1
  37. 2021Visualizing the embedding space to explain the effect of knowledge distillationcited by 0
  38. 2024Adaptive Teaching with Shared Classifier for Knowledge Distillationcited by 0

50 most cited of 69 All 69 (JSON)