# Online Knowledge Distillation via Multi-branch Diversity Enhancement

arXiv: https://arxiv.org/abs/2010.00795 · HTML: https://mlsanity.com/paper/2010.00795/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [PromptKD: Unsupervised Prompt Distillation for Vision-Language Models](https://mlsanity.com/paper/2403.02781/) (2024)
- [Online Knowledge Distillation for Efficient Pose Estimation](https://mlsanity.com/paper/2108.02092/) (2021)
- [Efficient Knowledge Distillation from Model Checkpoints](https://mlsanity.com/paper/2210.06458/) (2022)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?](https://mlsanity.com/paper/2305.12954/) (2023)
- [How to Train the Teacher Model for Effective Knowledge Distillation](https://mlsanity.com/paper/2407.18041/) (2024)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing](https://mlsanity.com/paper/2204.06625/) (2022)
- [Learn From the Past: Experience Ensemble Knowledge Distillation](https://mlsanity.com/paper/2202.12488/) (2022)
- [Improved Knowledge Distillation via Adversarial Collaboration](https://mlsanity.com/paper/2111.14356/) (2021)
- [Knowledge Distillation with Deep Supervision](https://mlsanity.com/paper/2202.07846/) (2022)
- [Low-Cost Self-Ensembles Based on Multi-Branch Transformation and Grouped Convolution](https://mlsanity.com/paper/2408.02307/) (2024)
- [Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity](https://mlsanity.com/paper/2510.22480/) (2025)
- [Beyond Student: An Asymmetric Network for Neural Network Inheritance](https://mlsanity.com/paper/2602.09509/) (2026)
- [Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation](https://mlsanity.com/paper/2110.00329/) (2021)
- [Decoupled Knowledge with Ensemble Learning for Online Distillation](https://mlsanity.com/paper/2312.11218/) (2023)
- [Estimating Human Poses Across Datasets: A Unified Skeleton and Multi-Teacher Distillation Approach](https://mlsanity.com/paper/2405.20084/) (2024)
- [Adaptive Teaching with Shared Classifier for Knowledge Distillation](https://mlsanity.com/paper/2406.08528/) (2024)
- [Classroom-Inspired Multi-Mentor Distillation with Adaptive Learning Strategies](https://mlsanity.com/paper/2409.20237/) (2024)

## Abstract (arXiv)

Knowledge distillation is an effective method to transfer the knowledge from the cumbersome teacher model to the lightweight student model. Online knowledge distillation uses the ensembled prediction results of multiple student models as soft targets to train each student model. However, the homogenization problem will lead to difficulty in further improving model performance. In this work, we propose a new distillation method to enhance the diversity among multiple student models. We introduce Feature Fusion Module (FFM), which improves the performance of the attention mechanism in the network by integrating rich semantic information contained in the last block of multiple student models. Furthermore, we use the Classifier Diversification(CD) loss function to strengthen the differences between the student models and deliver a better ensemble result. Extensive experiments proved that our method significantly enhances the diversity among student models and brings better distillation performance. We evaluate our method on three image classification datasets: CIFAR-10/100 and CINIC-10. The results show that our method achieves state-of-the-art performance on these datasets.