# Peer Collaborative Learning for Online Knowledge Distillation

arXiv: https://arxiv.org/abs/2006.04147 · HTML: https://mlsanity.com/paper/2006.04147/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 30 of 58)

- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021)
- [Learning Student-Friendly Teacher Networks for Knowledge Distillation](https://mlsanity.com/paper/2102.07650/) (2021)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [Mutual Contrastive Learning for Visual Representation Learning](https://mlsanity.com/paper/2104.12565/) (2021)
- [Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition](https://mlsanity.com/paper/2207.11518/) (2022)
- [A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation](https://mlsanity.com/paper/2307.12574/) (2023)
- [Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning](https://mlsanity.com/paper/2309.05911/) (2023)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation](https://mlsanity.com/paper/2203.03910/) (2022)
- [Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2503.16707/) (2025)
- [Experts Weights Averaging: A New General Training Scheme for Vision Transformers](https://mlsanity.com/paper/2308.06093/) (2023)
- [MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts](https://mlsanity.com/paper/2402.00893/) (2024)
- [AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Models](https://mlsanity.com/paper/2405.16240/) (2024)
- [ScaleKD: Strong Vision Transformers Could Be Excellent Teachers](https://mlsanity.com/paper/2411.06786/) (2024)
- [Delta Distillation for Efficient Video Processing](https://mlsanity.com/paper/2203.09594/) (2022)
- [Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge](https://mlsanity.com/paper/2106.01489/) (2021)
- [Knowledge Distillation with Refined Logits](https://mlsanity.com/paper/2408.07703/) (2024)
- [Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation](https://mlsanity.com/paper/2303.14666/) (2023)
- [DynamicKD: An Effective Knowledge Distillation via Dynamic Entropy Correction-Based Distillation for Gap Optimizing](https://mlsanity.com/paper/2305.05233/) (2023)
- [PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor](https://mlsanity.com/paper/2403.06668/) (2024)
- [CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing](https://mlsanity.com/paper/2204.06625/) (2022)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation](https://mlsanity.com/paper/2408.13698/) (2024)
- [Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filtering](https://mlsanity.com/paper/2202.13140/) (2022)
- [Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2507.10348/) (2025)

## Abstract (arXiv)

Traditional knowledge distillation uses a two-stage training strategy to transfer knowledge from a high-capacity teacher model to a compact student model, which relies heavily on the pre-trained teacher. Recent online knowledge distillation alleviates this limitation by collaborative learning, mutual learning and online ensembling, following a one-stage end-to-end training fashion. However, collaborative learning and mutual learning fail to construct an online high-capacity teacher, whilst online ensembling ignores the collaboration among branches and its logit summation impedes the further optimisation of the ensemble teacher. In this work, we propose a novel Peer Collaborative Learning method for online knowledge distillation, which integrates online ensembling and network collaboration into a unified framework. Specifically, given a target network, we construct a multi-branch network for training, in which each branch is called a peer. We perform random augmentation multiple times on the inputs to peers and assemble feature representations outputted from peers with an additional classifier as the peer ensemble teacher. This helps to transfer knowledge from a high-capacity teacher to peers, and in turn further optimises the ensemble teacher. Meanwhile, we employ the temporal mean model of each peer as the peer mean teacher to collaboratively transfer knowledge among peers, which helps each peer to learn richer knowledge and facilitates to optimise a more stable model with better generalisation. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet show that the proposed method significantly improves the generalisation of various backbone networks and outperforms the state-of-the-art methods.