# Contrastive Regularization for Semi-Supervised Learning

arXiv: https://arxiv.org/abs/2201.06247 · HTML: https://mlsanity.com/paper/2201.06247/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [CoMatch: Semi-supervised Learning with Contrastive Graph Regularization](https://mlsanity.com/paper/2011.11183/) (2020)
- [Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning](https://mlsanity.com/paper/2007.11330/) (2020)
- [SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06480/) (2021)
- [On Data-Augmentation and Consistency-Based Semi-Supervised Learning](https://mlsanity.com/paper/2101.06967/) (2021)

## Built on by (most cited first, 24 of 24)

- [The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning](https://mlsanity.com/paper/2303.00106/) (2023)
- [Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data](https://mlsanity.com/paper/2303.11066/) (2023)
- [ConMatch: Semi-Supervised Learning with Confidence-Guided Consistency Regularization](https://mlsanity.com/paper/2208.08631/) (2022)
- [Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning](https://mlsanity.com/paper/2405.11756/) (2024)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students](https://mlsanity.com/paper/2209.02178/) (2022)
- [InstanT: Semi-supervised Learning with Instance-dependent Thresholds](https://mlsanity.com/paper/2310.18910/) (2023)
- [Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators](https://mlsanity.com/paper/2307.05358/) (2023)
- [Robust Semi-Supervised Learning for Self-learning Open-World Classes](https://mlsanity.com/paper/2401.07551/) (2024)
- [Towards Unbiased Training in Federated Open-world Semi-supervised Learning](https://mlsanity.com/paper/2305.00771/) (2023)
- [SPLAL: Similarity-based pseudo-labeling with alignment loss for semi-supervised medical image classification](https://mlsanity.com/paper/2307.04610/) (2023)
- [DualMatch: Robust Semi-Supervised Learning with Dual-Level Interaction](https://mlsanity.com/paper/2310.16459/) (2023)
- [Active Semi-Supervised Learning by Exploring Per-Sample Uncertainty and Consistency](https://mlsanity.com/paper/2303.08978/) (2023)
- [Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data](https://mlsanity.com/paper/2306.01222/) (2023)
- [ContrasInver: Ultra-Sparse Label Semi-supervised Regression for Multi-dimensional Seismic Inversion](https://mlsanity.com/paper/2302.06441/) (2023)
- [A Method of Moments Embedding Constraint and its Application to Semi-Supervised Learning](https://mlsanity.com/paper/2404.17978/) (2024)
- [Investigating the Impact of Semi-Supervised Methods with Data Augmentation on Offensive Language Detection in Romanian Language](https://mlsanity.com/paper/2407.20076/) (2024)
- [SemiSFL: Split Federated Learning on Unlabeled and Non-IID Data](https://mlsanity.com/paper/2307.15870/) (2023)
- [Rethinking Scale Imbalance in Semi-supervised Object Detection for Aerial Images](https://mlsanity.com/paper/2310.14718/) (2023)
- [Smooth Pseudo-Labeling](https://mlsanity.com/paper/2405.14313/) (2024)
- [Domain-Agnostic Feature Modulation for Semi-Supervised Domain Generalization](https://mlsanity.com/paper/2503.20897/) (2025)
- [Predicting butterfly species presence from satellite imagery using soft contrastive regularisation](https://mlsanity.com/paper/2505.09306/) (2025)
- [NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model](https://mlsanity.com/paper/2602.06093/) (2026)
- [How to Achieve Prototypical Birth and Death for OOD Detection?](https://mlsanity.com/paper/2603.15650/) (2026)

## Abstract (arXiv)

Consistency regularization on label predictions becomes a fundamental technique in semi-supervised learning, but it still requires a large number of training iterations for high performance. In this study, we analyze that the consistency regularization restricts the propagation of labeling information due to the exclusion of samples with unconfident pseudo-labels in the model updates. Then, we propose contrastive regularization to improve both efficiency and accuracy of the consistency regularization by well-clustered features of unlabeled data. In specific, after strongly augmented samples are assigned to clusters by their pseudo-labels, our contrastive regularization updates the model so that the features with confident pseudo-labels aggregate the features in the same cluster, while pushing away features in different clusters. As a result, the information of confident pseudo-labels can be effectively propagated into more unlabeled samples during training by the well-clustered features. On benchmarks of semi-supervised learning tasks, our contrastive regularization improves the previous consistency-based methods and achieves state-of-the-art results, especially with fewer training iterations. Our method also shows robust performance on open-set semi-supervised learning where unlabeled data includes out-of-distribution samples.