# Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression

arXiv: https://arxiv.org/abs/2110.11023 · HTML: https://mlsanity.com/paper/2110.11023/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 8 of 8)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation](https://mlsanity.com/paper/2010.01532/) (2020)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data](https://mlsanity.com/paper/2104.03618/) (2021)

## Built on by (most cited first, 3 of 3)

- [PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor](https://mlsanity.com/paper/2403.06668/) (2024)
- [Knowledge distillation with a class-aware loss for endoscopic disease detection](https://mlsanity.com/paper/2207.09530/) (2022)
- [Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging](https://mlsanity.com/paper/2212.02931/) (2022)

## Abstract (arXiv)

Knowledge distillation (KD) is an effective model compression technique where a compact student network is taught to mimic the behavior of a complex and highly trained teacher network. In contrast, Mutual Learning (ML) provides an alternative strategy where multiple simple student networks benefit from sharing knowledge, even in the absence of a powerful but static teacher network. Motivated by these findings, we propose a single-teacher, multi-student framework that leverages both KD and ML to achieve better performance. Furthermore, an online distillation strategy is utilized to train the teacher and students simultaneously. To evaluate the performance of the proposed approach, extensive experiments were conducted using three different versions of teacher-student networks on benchmark biomedical classification (MSI vs. MSS) and object detection (Polyp Detection) tasks. Ensemble of student networks trained in the proposed manner achieved better results than the ensemble of students trained using KD or ML individually, establishing the benefit of augmenting knowledge transfer from teacher to students with peer-to-peer learning between students.