# Revisiting Consistency Regularization for Semi-Supervised Learning

arXiv: https://arxiv.org/abs/2112.05825 · HTML: https://mlsanity.com/paper/2112.05825/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 78 references that are in the map.
- First posted on arXiv 10 Dec 2021.

## Builds on (most cited first, 30 of 54)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)

## Built on by (most cited first, 30 of 34)

- [SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning](https://mlsanity.com/paper/2301.10921/) (2023)
- [USB: A Unified Semi-supervised Learning Benchmark for Classification](https://mlsanity.com/paper/2208.07204/) (2022)
- [Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception?](https://mlsanity.com/paper/2312.04548/) (2023)
- [Temporally Consistent Koopman Autoencoders for Forecasting Dynamical Systems](https://mlsanity.com/paper/2403.12335/) (2024)
- [SemiReward: A General Reward Model for Semi-supervised Learning](https://mlsanity.com/paper/2310.03013/) (2023)
- [An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2211.11086/) (2022)
- [SimMatchV2: Semi-Supervised Learning with Graph Consistency](https://mlsanity.com/paper/2308.06692/) (2023)
- [CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation](https://mlsanity.com/paper/2503.18244/) (2025)
- [SSB: Simple but Strong Baseline for Boosting Performance of Open-Set Semi-Supervised Learning](https://mlsanity.com/paper/2311.10572/) (2023)
- [Joint-Task Regularization for Partially Labeled Multi-Task Learning](https://mlsanity.com/paper/2404.01976/) (2024)
- [Boosting Semi-Supervised Learning with Contrastive Complementary Labeling](https://mlsanity.com/paper/2212.06643/) (2022)
- [Reliability-Adaptive Consistency Regularization for Weakly-Supervised Point Cloud Segmentation](https://mlsanity.com/paper/2303.05164/) (2023)
- [DisCo: Distilled Student Models Co-training for Semi-supervised Text Mining](https://mlsanity.com/paper/2305.12074/) (2023)
- [FAGC:Feature Augmentation on Geodesic Curve in the Pre-Shape Space](https://mlsanity.com/paper/2312.03325/) (2023)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [Semi-MoE: Mixture-of-Experts meets Semi-Supervised Histopathology Segmentation](https://mlsanity.com/paper/2509.13834/) (2025)
- [S5: Scalable Semi-Supervised Semantic Segmentation in Remote Sensing](https://mlsanity.com/paper/2508.12409/) (2025)
- [Integration of Old and New Knowledge for Generalized Intent Discovery: A Consistency-driven Prototype-Prompting Framework](https://mlsanity.com/paper/2506.08490/) (2025)
- [Multimodal Prototype Alignment for Semi-supervised Pathology Image Segmentation](https://mlsanity.com/paper/2508.19574/) (2025)
- [Pruning the Unlabeled Data to Improve Semi-Supervised Learning](https://mlsanity.com/paper/2308.14058/) (2023)
- [MaskMatch: Boosting Semi-Supervised Learning Through Mask Autoencoder-Driven Feature Learning](https://mlsanity.com/paper/2405.06227/) (2024)
- [RS-MTDF: Multi-Teacher Distillation and Fusion for Remote Sensing Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2506.08772/) (2025)
- [BARL: Bilateral Alignment in Representation and Label Spaces for Semi-Supervised Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2510.16863/) (2025)
- [Towards Generalizable Morph Attack Detection with Consistency Regularization](https://mlsanity.com/paper/2308.10392/) (2023)
- [Feature Space Renormalization for Semi-supervised Learning](https://mlsanity.com/paper/2311.04055/) (2023)
- [Boosting Semi-Supervised 2D Human Pose Estimation by Revisiting Data Augmentation and Consistency Training](https://mlsanity.com/paper/2402.11566/) (2024)
- [SeMi: When Imbalanced Semi-Supervised Learning Meets Mining Hard Examples](https://mlsanity.com/paper/2501.06004/) (2025)
- [CRUPL: A Semi-Supervised Cyber Attack Detection with Consistency Regularization and Uncertainty-aware Pseudo-Labeling in Smart Grid](https://mlsanity.com/paper/2503.00358/) (2025)
- [Dual Teacher-Student Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2505.11018/) (2025)
- [U-Mamba2-SSL for Semi-Supervised Tooth and Pulp Segmentation in CBCT](https://mlsanity.com/paper/2509.20154/) (2025)

## Abstract (arXiv)

Consistency regularization is one of the most widely-used techniques for semi-supervised learning (SSL). Generally, the aim is to train a model that is invariant to various data augmentations. In this paper, we revisit this idea and find that enforcing invariance by decreasing distances between features from differently augmented images leads to improved performance. However, encouraging equivariance instead, by increasing the feature distance, further improves performance. To this end, we propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss, that imposes consistency and equivariance on the classifier and the feature level, respectively. Experimental results show that our model defines a new state of the art for various datasets and settings and outperforms previous work by a significant margin, particularly in low data regimes. Extensive experiments are conducted to analyze the method, and the code will be published.