# Confidence-Aware Multi-Teacher Knowledge Distillation arXiv: https://arxiv.org/abs/2201.00007 ยท HTML: https://mlsanity.com/paper/2201.00007/ ## Key facts - Cited by 39 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 13 of its 20 references that are in the map. - First posted on arXiv 30 Dec 2021. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 13 of 13) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013) - [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019) - [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018) - [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020) - [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020) - [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019) - [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021) ## Built on by (most cited first, 30 of 39) - [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022) - [Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning](https://mlsanity.com/paper/2306.06634/) (2023) - [Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition](https://mlsanity.com/paper/2502.18510/) (2025) - [Knowledge Distillation with Refined Logits](https://mlsanity.com/paper/2408.07703/) (2024) - [LibEER: A Comprehensive Benchmark and Algorithm Library for EEG-based Emotion Recognition](https://mlsanity.com/paper/2410.09767/) (2024) - [Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty](https://mlsanity.com/paper/2305.02722/) (2023) - [Progressive Cross-modal Knowledge Distillation for Human Action Recognition](https://mlsanity.com/paper/2208.08090/) (2022) - [Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks](https://mlsanity.com/paper/2205.02468/) (2022) - [TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration](https://mlsanity.com/paper/2410.12183/) (2024) - [Adaptive Group Robust Ensemble Knowledge Distillation](https://mlsanity.com/paper/2411.14984/) (2024) - [Label driven Knowledge Distillation for Federated Learning with non-IID Data](https://mlsanity.com/paper/2209.14520/) (2022) - [Double Reverse Regularization Network Based on Self-Knowledge Distillation for SAR Object Classification](https://mlsanity.com/paper/2311.15231/) (2023) - [Distilling Calibrated Student from an Uncalibrated Teacher](https://mlsanity.com/paper/2302.11472/) (2023) - [MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition](https://mlsanity.com/paper/2408.16563/) (2024) - [Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition](https://mlsanity.com/paper/2506.08717/) (2025) - [Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer](https://mlsanity.com/paper/2509.20854/) (2025) - [Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision](https://mlsanity.com/paper/2210.13743/) (2022) - [CST: Calibration Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2402.12736/) (2024) - [GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation](https://mlsanity.com/paper/2405.03764/) (2024) - [Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detection](https://mlsanity.com/paper/2407.01894/) (2024) - [Distilled-3DGS:Distilled 3D Gaussian Splatting](https://mlsanity.com/paper/2508.14037/) (2025) - [Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors](https://mlsanity.com/paper/2605.27967/) (2026) - [Regime-Aware Peer Specialization for Robust RAG under Heterogeneous Knowledge Conflicts](https://mlsanity.com/paper/2606.30518/) (2026) - [CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning](https://mlsanity.com/paper/2609.00446/) (2026) - [Fuzzy Knowledge Distillation from High-Order TSK to Low-Order TSK](https://mlsanity.com/paper/2302.08038/) (2023) - [Education distillation:getting student models to learn in shcools](https://mlsanity.com/paper/2311.13811/) (2023) - [Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data](https://mlsanity.com/paper/2407.05315/) (2024) - [Classroom-Inspired Multi-Mentor Distillation with Adaptive Learning Strategies](https://mlsanity.com/paper/2409.20237/) (2024) - [MIDAS: Modeling Ground-Truth Distributions with Dark Knowledge for Domain Generalized Stereo Matching](https://mlsanity.com/paper/2503.04376/) (2025) - [Collaborative Unlabeled Data Optimization](https://mlsanity.com/paper/2505.14117/) (2025) ## Abstract (arXiv) Knowledge distillation is initially introduced to utilize additional supervision from a single teacher model for the student model training. To boost the student performance, some recent variants attempt to exploit diverse knowledge sources from multiple teachers. However, existing studies mainly integrate knowledge from diverse sources by averaging over multiple teacher predictions or combining them using other various label-free strategies, which may mislead student in the presence of low-quality teacher predictions. To tackle this problem, we propose Confidence-Aware Multi-teacher Knowledge Distillation (CA-MKD), which adaptively assigns sample-wise reliability for each teacher prediction with the help of ground-truth labels, with those teacher predictions close to one-hot labels assigned large weights. Besides, CA-MKD incorporates intermediate layers to stable the knowledge transfer process. Extensive experiments show that our CA-MKD consistently outperforms all compared state-of-the-art methods across various teacher-student architectures.