mlsanityData to 14 Oct 2026

Confidence-Aware Multi-Teacher Knowledge Distillation

arXiv 2201.00007 ↗First posted 30 Dec 2021cs.LG cs.AIcited by 39

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation is initially introduced to utilize additional supervision from a single teacher model for the student model training. To boost the student performance, some recent variants attempt to exploit diverse knowledge sources from multiple teachers. However, existing studies mainly integrate knowledge from diverse sources by averaging over multiple teacher predictions or combining them using other various label-free strategies, which may mislead student in the presence of low-quality teacher predictions. To tackle this problem, we propose Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD), which adaptively assigns sample-wise reliability for each teacher prediction with the help of ground-truth labels, with those teacher predictions close to one-hot labels assigned large weights. Besides, CA-MKD incorporates intermediate layers to stable the knowledge transfer process. Extensive experiments show that our CA-MKD consistently outperforms all compared state-of-the-art methods across various teacher-student architectures.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 13 of its 20 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

13 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation6 Mar

Other references · 12

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  6. 2019Contrastive Representation Distillationcited by 748
  7. 2013Do Deep Nets Really Need to be Deep?cited by 474
  8. 2019Variational Information Distillation for Knowledge Transfercited by 324
  9. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  10. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  11. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  12. 2019Online Knowledge Distillation with Diverse Peerscited by 128

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 6; 2024: 10; 2025: 7; 2026: 11.

39 papers.

Sort:

By area:

  1. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
  2. 2023Adaptive Multi-Teacher Knowledge Distillation with Meta-Learningcited by 16
  3. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  4. 2024Knowledge Distillation with Refined Logitscited by 9
  5. 2024LibEER: A Comprehensive Benchmark and Algorithm Library for EEG-based Emotion Recognitioncited by 8
  6. 2023Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertaintycited by 7
  7. 2022Progressive Cross-modal Knowledge Distillation for Human Action Recognitioncited by 6
  8. 2022Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networkscited by 5
  9. 2024TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaborationcited by 5
  10. 2024Adaptive Group Robust Ensemble Knowledge Distillationcited by 5
  11. 2022Label driven Knowledge Distillation for Federated Learning with non-IID Datacited by 3
  12. 2023Double Reverse Regularization Network Based on Self-Knowledge Distillation for SAR Object Classificationcited by 3
27 more
  1. 2023Distilling Calibrated Student from an Uncalibrated Teachercited by 2
  2. 2024MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognitioncited by 2
  3. 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognitioncited by 2
  4. 2025Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizercited by 2
  5. 2022Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervisioncited by 1
  6. 2024CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 1
  7. 2024GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillationcited by 1
  8. 2024Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detectioncited by 1
  9. 2025Distilled-3DGS:Distilled 3D Gaussian Splattingcited by 1
  10. 2026Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priorscited by 1
  11. 2026Regime-Aware Peer Specialization for Robust RAG under Heterogeneous Knowledge Conflictscited by 1
  12. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learningcited by 1
  13. 2023Fuzzy Knowledge Distillation from High-Order TSK to Low-Order TSKcited by 0
  14. 2023Education distillation:getting student models to learn in shcoolscited by 0
  15. 2024Topological Persistence Guided Knowledge Distillation for Wearable Sensor Datacited by 0
  16. 2024Classroom-Inspired Multi-Mentor Distillation with Adaptive Learning Strategiescited by 0
  17. 2025MIDAS: Modeling Ground-Truth Distributions with Dark Knowledge for Domain Generalized Stereo Matchingcited by 0
  18. 2025Collaborative Unlabeled Data Optimizationcited by 0
  19. 2025Optimally Deep Networks -- Adapting Model Depth to Datasets for Superior Efficiencycited by 0
  20. 2026REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiencycited by 0
  21. 2026A Benchmark Study of Neural Network Compression Methods for Hyperspectral Image Classificationcited by 0
  22. 2026Efficient INT8 Single-Image Super-Resolution via Deployment-Aware Quantization and Teacher-Guided Trainingcited by 0
  23. 2026UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learningcited by 0
  24. 2026Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentationcited by 0
  25. 2026CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learningcited by 0
  26. 2026Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detectioncited by 0
  27. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework
  2. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation
  3. 2026Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
  4. 2022Unified and Effective Ensemble Knowledge Distillation
  5. 2022Parameter-Efficient and Student-Friendly Knowledge Distillation
  6. 2022Toward Student-Oriented Teacher Network Training For Knowledge Distillation
  7. 2022Knowledge Condensation Distillation
  8. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios