# Semi-Supervised Learning with Multi-Head Co-Training

arXiv: https://arxiv.org/abs/2107.04795 · HTML: https://mlsanity.com/paper/2107.04795/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 10 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [CoMatch: Semi-supervised Learning with Contrastive Graph Regularization](https://mlsanity.com/paper/2011.11183/) (2020)
- [Deep Co-Training for Semi-Supervised Image Recognition](https://mlsanity.com/paper/1803.05984/) (2018)
- [There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average](https://mlsanity.com/paper/1806.05594/) (2018)
- [Adversarial Dropout for Supervised and Semi-supervised Learning](https://mlsanity.com/paper/1707.03631/) (2017)
- [A Survey on Semi-Supervised Learning Techniques](https://mlsanity.com/paper/1402.4645/) (2014)
- [Adversarial Transformations for Semi-Supervised Learning](https://mlsanity.com/paper/1911.06181/) (2019)
- [Theoretical Foundation of Co-Training and Disagreement-Based Algorithms](https://mlsanity.com/paper/1708.04403/) (2017)

## Built on by (most cited first, 15 of 15)

- [SimMatch: Semi-supervised Learning with Similarity Matching](https://mlsanity.com/paper/2203.06915/) (2022)
- [A Review of Pseudo-Labeling for Computer Vision](https://mlsanity.com/paper/2408.07221/) (2024)
- [Estimating Soft Labels for Out-of-Domain Intent Detection](https://mlsanity.com/paper/2211.05561/) (2022)
- [Boosting Semi-Supervised Learning by bridging high and low-confidence predictions](https://mlsanity.com/paper/2308.07509/) (2023)
- [SequenceMatch: Revisiting the design of weak-strong augmentations for Semi-supervised learning](https://mlsanity.com/paper/2310.15787/) (2023)
- [OTMatch: Improving Semi-Supervised Learning with Optimal Transport](https://mlsanity.com/paper/2310.17455/) (2023)
- [Debiasing, calibrating, and improving Semi-supervised Learning performance via simple Ensemble Projector](https://mlsanity.com/paper/2310.15764/) (2023)
- [Self-training through Classifier Disagreement for Cross-Domain Opinion Target Extraction](https://mlsanity.com/paper/2302.14719/) (2023)
- [Semi-Supervised Medical Image Segmentation with Co-Distribution Alignment](https://mlsanity.com/paper/2307.12630/) (2023)
- [MultiMatch: Multihead Consistency Regularization Matching for Semi-Supervised Text Classification](https://mlsanity.com/paper/2506.07801/) (2025)
- [CATCHFed: Efficient Unlabeled Data Utilization for Semi-Supervised Federated Learning in Limited Labels Environments](https://mlsanity.com/paper/2511.11778/) (2025)
- [LayerMatch: Do Pseudo-labels Benefit All Layers?](https://mlsanity.com/paper/2406.14207/) (2024)
- [SpikeMatch: Semi-Supervised Learning with Temporal Dynamics of Spiking Neural Networks](https://mlsanity.com/paper/2509.22581/) (2025)
- [Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data](https://mlsanity.com/paper/2601.20072/) (2026)
- [From Mobile Data to Business Insights: An End-to-End Analytics Framework for Large-Scale Urban Mobility Analysis and Decision Support](https://mlsanity.com/paper/2607.03394/) (2026)

## Abstract (arXiv)

Co-training, extended from self-training, is one of the frameworks for semi-supervised learning. Without natural split of features, single-view co-training works at the cost of training extra classifiers, where the algorithm should be delicately designed to prevent individual classifiers from collapsing into each other. To remove these obstacles which deter the adoption of single-view co-training, we present a simple and efficient algorithm Multi-Head Co-Training. By integrating base learners into a multi-head structure, the model is in a minimal amount of extra parameters. Every classification head in the unified model interacts with its peers through a "Weak and Strong Augmentation" strategy, in which the diversity is naturally brought by the strong data augmentation. Therefore, the proposed method facilitates single-view co-training by 1). promoting diversity implicitly and 2). only requiring a small extra computational overhead. The effectiveness of Multi-Head Co-Training is demonstrated in an empirical study on standard semi-supervised learning benchmarks.