# Collaborative Learning for Deep Neural Networks

arXiv: https://arxiv.org/abs/1805.11761 · HTML: https://mlsanity.com/paper/1805.11761/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2018.

## Built on by (most cited first, 30 of 69)

- [Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution](https://mlsanity.com/paper/2204.13062/) (2022)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation](https://mlsanity.com/paper/2209.04599/) (2022)
- [Mutual Contrastive Learning for Visual Representation Learning](https://mlsanity.com/paper/2104.12565/) (2021)
- [Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition](https://mlsanity.com/paper/2207.11518/) (2022)
- [A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation](https://mlsanity.com/paper/2307.12574/) (2023)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning](https://mlsanity.com/paper/2309.05911/) (2023)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [On the Reproducibility of Neural Network Predictions](https://mlsanity.com/paper/2102.03349/) (2021)
- [Switchable Online Knowledge Distillation](https://mlsanity.com/paper/2209.04996/) (2022)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Improving Plasticity in Online Continual Learning via Collaborative Learning](https://mlsanity.com/paper/2312.00600/) (2023)
- [Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation](https://mlsanity.com/paper/2203.03910/) (2022)
- [Mutual Adversarial Training: Learning together is better than going alone](https://mlsanity.com/paper/2112.05005/) (2021)
- [On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification](https://mlsanity.com/paper/2207.13975/) (2022)
- [Revisiting Knowledge Distillation: An Inheritance and Exploration Framework](https://mlsanity.com/paper/2107.00181/) (2021)
- [Predictive Churn with the Set of Good Models](https://mlsanity.com/paper/2402.07745/) (2024)
- [I3CL:Intra- and Inter-Instance Collaborative Learning for Arbitrary-shaped Scene Text Detection](https://mlsanity.com/paper/2108.01343/) (2021)
- [Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution](https://mlsanity.com/paper/2109.03075/) (2021)
- [Recurrent Early Exits for Federated Learning with Heterogeneous Clients](https://mlsanity.com/paper/2405.14791/) (2024)
- [Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation](https://mlsanity.com/paper/2311.08640/) (2023)
- [Vulnerability-Aware Spatio-Temporal Learning for Generalizable Deepfake Video Detection](https://mlsanity.com/paper/2501.01184/) (2025)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [Latent Class-Conditional Noise Model](https://mlsanity.com/paper/2302.09595/) (2023)
- [Co-training 2L Submodels for Visual Recognition](https://mlsanity.com/paper/2212.04884/) (2022)
- [Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles](https://mlsanity.com/paper/2412.15758/) (2024)
- [Locally Adaptive Label Smoothing for Predictive Churn](https://mlsanity.com/paper/2102.05140/) (2021)

## Abstract (arXiv)

We introduce collaborative learning in which multiple classifier heads of the same network are simultaneously trained on the same training data to improve generalization and robustness to label noise with no extra inference cost. It acquires the strengths from auxiliary training, multi-task learning and knowledge distillation. There are two important mechanisms involved in collaborative learning. First, the consensus of multiple views from different classifier heads on the same example provides supplementary information as well as regularization to each classifier, thereby improving generalization. Second, intermediate-level representation (ILR) sharing with backpropagation rescaling aggregates the gradient flows from all heads, which not only reduces training computational complexity, but also facilitates supervision to the shared layers. The empirical results on CIFAR and ImageNet datasets demonstrate that deep neural networks learned as a group in a collaborative way significantly reduce the generalization error and increase the robustness to label noise.