mlsanityData to 7 Oct 2026

Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge

arXiv 2106.01489 ↗First posted 2 Jun 2021cs.LG cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Mutual knowledge distillation (MKD) improves a model by distilling knowledge from another model. However, textitnot all knowledge is certain and correct, especially under adverse conditions. For example, label noise usually leads to less reliable models due to undesired memorization citezhang2017understanding,arpit2017closer. Wrong knowledge misleads the learning rather than helps. This problem can be handled by two aspects: (i) improving the reliability of a model where the knowledge is from (i.e., knowledge source's reliability); (ii) selecting reliable knowledge for distillation. In the literature, making a model more reliable is widely studied while selective MKD receives little attention. Therefore, we focus on studying selective MKD. Concretely, a generic MKD framework, underlineConfident knowledge selection followed by underlineMutual underlineDistillation (CMD), is designed. The key component of CMD is a generic knowledge selection formulation, making the selection threshold either static (CMD-S) or progressive (CMD-P). Additionally, CMD covers two special cases: zero-knowledge and all knowledge, leading to a unified MKD framework. Extensive experiments are present to demonstrate the effectiveness of CMD and thoroughly justify the design of CMD. For example, CMD-P obtains new state-of-the-art results in robustness against label noise.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

31 of 45 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  6. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  7. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  8. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  9. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  10. 2019When Does Label Smoothing Help?cited by 985
  11. 2017A Closer Look at Memorization in Deep Networkscited by 936
  12. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
19 more
  1. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  2. 2017Deep Mutual Learningcited by 621
  3. 2018Born Again Neural Networkscited by 539
  4. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  5. 2013Do Deep Nets Really Need to be Deep?cited by 466
  6. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  7. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  8. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  9. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  10. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  11. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  12. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  13. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  14. 2020Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learningcited by 142
  15. 2020Feature-map-level Online Adversarial Knowledge Distillationcited by 62
  16. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  17. 2019Feature Fusion for Online Mutual Knowledge Distillationcited by 40
  18. 2019IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matterscited by 33
  19. 2020ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networkscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 1; 2024: 1; 2025: 4; 2026: 1.

9 papers.

  1. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  2. 2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
  3. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  4. 2021Data Selection for Efficient Model Update in Federated Learningcited by 2
  5. 2025Emergent Specialization: Rare Token Neurons in Language Modelscited by 2
  6. 2024KnFu: Effective Knowledge Fusioncited by 1
  7. 2025Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Modelscited by 0
  8. 2025Distributed Specialization: Rare-Token Neurons in Large Language Modelscited by 0
  9. 2026FedCC: Towards Addressing Label Distribution Skews in Distillation-Based Federated Learningcited by 0