mlsanityData to 7 Oct 2026

Decoupled Knowledge with Ensemble Learning for Online Distillation

arXiv 2312.11218 ↗First posted 18 Dec 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Offline distillation is a two-stage pipeline that requires expensive resources to train a teacher network and then distill the knowledge to a student for deployment. Online knowledge distillation, on the other hand, is a one-stage strategy that alleviates the requirement with mutual learning and collaborative learning. Recent peer collaborative learning (PCL) integrates online ensemble, collaboration of base networks and temporal mean teacher to construct effective knowledge. However, the model collapses occasionally in PCL due to high homogenization between the student and the teacher. In this paper, the cause of the high homogenization is analyzed and the solution is presented. A decoupled knowledge for online knowledge distillation is generated by an independent teacher, separate from the student. Such design can increase the diversity between the networks and reduce the possibility of model collapse. To obtain early decoupled knowledge, an initialization scheme for the teacher is devised, and a 2D geometry-based analysis experiment is conducted under ideal conditions to showcase the effectiveness of this scheme. Moreover, to improve the teacher's supervisory resilience, a decaying ensemble scheme is devised. It assembles the knowledge of the teacher to which a dynamic weight which is large at the start of the training and gradually decreases with the training process is assigned. The assembled knowledge serves as a strong teacher during the early training and the decreased-weight-assembled knowledge can eliminate the distribution deviation under the potentially overfitted teacher's supervision. A Monte Carlo-based simulation is conducted to evaluate the convergence. Extensive experiments on CIFAR-10, CIFAR-100 and TinyImageNet show the superiority of our method. Ablation studies and further analysis demonstrate the effectiveness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

21 of 33 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016Wide Residual Networkscited by 3,057
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2020Knowledge Distillation: A Surveycited by 1,578
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  9. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  10. 2019Relational Knowledge Distillationcited by 873
  11. 2017Deep Mutual Learningcited by 621
  12. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
9 more
  1. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  2. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  3. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  4. 2021A Survey of Modern Deep Learning based Object Detection Modelscited by 97
  5. 2018Collaborative Learning for Deep Neural Networkscited by 69
  6. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  7. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  8. 2020Online Knowledge Distillation via Multi-branch Diversity Enhancementcited by 21
  9. 2021Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distributioncited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Group Fairness in Knowledge Distillation via Laplace Approximation of Early Exits
  2. 2025Asymmetric Decision-Making in Online Knowledge Distillation:Unifying Consensus and Divergence
  3. 2026PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
  4. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review
  5. 2025Knowledge Distillation with Training Wheels
  6. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failures
  7. 2026Distribution Corrected Offline Data Distillation for Large Language Models
  8. 2026Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall