# SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning

arXiv: https://arxiv.org/abs/2101.06480 · HTML: https://mlsanity.com/paper/2101.06480/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 16 Jan 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)

## Built on by (most cited first, 30 of 35)

- [Multi-Layer Pseudo-Supervision for Histopathology Tissue Semantic Segmentation using Patch-level Classification Labels](https://mlsanity.com/paper/2110.08048/) (2021)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding](https://mlsanity.com/paper/2209.06638/) (2022)
- [Boosting Semi-Supervised Learning by Exploiting All Unlabeled Data](https://mlsanity.com/paper/2303.11066/) (2023)
- [ConMatch: Semi-Supervised Learning with Confidence-Guided Consistency Regularization](https://mlsanity.com/paper/2208.08631/) (2022)
- [Contrastive Regularization for Semi-Supervised Learning](https://mlsanity.com/paper/2201.06247/) (2022)
- [Image Data Augmentation Approaches: A Comprehensive Survey and Future directions](https://mlsanity.com/paper/2301.02830/) (2023)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Barely-Supervised Learning: Semi-Supervised Learning with very few labeled images](https://mlsanity.com/paper/2112.12004/) (2021)
- [Robust Mutual Learning for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.00609/) (2021)
- [A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clustering](https://mlsanity.com/paper/2106.16209/) (2021)
- [Systematic comparison of semi-supervised and self-supervised learning for medical image classification](https://mlsanity.com/paper/2307.08919/) (2023)
- [Unsupervised Embedding Adaptation via Early-Stage Feature Reconstruction for Few-Shot Classification](https://mlsanity.com/paper/2106.11486/) (2021)
- [ActiveMatch: End-to-end Semi-supervised Active Representation Learning](https://mlsanity.com/paper/2110.02521/) (2021)
- [FixMatchSeg: Fixing FixMatch for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.00400/) (2022)
- [Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection](https://mlsanity.com/paper/2207.11789/) (2022)
- [Boosting Semi-Supervised Learning with Contrastive Complementary Labeling](https://mlsanity.com/paper/2212.06643/) (2022)
- [Graph Convolutional Networks based on Manifold Learning for Semi-Supervised Image Classification](https://mlsanity.com/paper/2304.12492/) (2023)
- [Active Self-Semi-Supervised Learning for Few Labeled Samples](https://mlsanity.com/paper/2203.04560/) (2022)
- [AuxMix: Semi-Supervised Learning with Unconstrained Unlabeled Data](https://mlsanity.com/paper/2206.06959/) (2022)
- [Robot Instance Segmentation with Few Annotations for Grasping](https://mlsanity.com/paper/2407.01302/) (2024)
- [Deep Reference Priors: What is the best way to pretrain a model?](https://mlsanity.com/paper/2202.00187/) (2022)
- [Semi-Supervised End-To-End Contrastive Learning For Time Series Classification](https://mlsanity.com/paper/2310.08848/) (2023)
- [Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)](https://mlsanity.com/paper/2403.18699/) (2024)
- [ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision](https://mlsanity.com/paper/2304.09369/) (2023)
- [FireMatch: A Semi-Supervised Video Fire Detection Network Based on Consistency and Distribution Alignment](https://mlsanity.com/paper/2311.05168/) (2023)
- [Bridging the Gaps: Utilizing Unlabeled Face Recognition Datasets to Boost Semi-Supervised Facial Expression Recognition](https://mlsanity.com/paper/2410.17622/) (2024)
- [Shuffle & Divide: Contrastive Learning for Long Text](https://mlsanity.com/paper/2304.09374/) (2023)
- [Rethinking Scale Imbalance in Semi-supervised Object Detection for Aerial Images](https://mlsanity.com/paper/2310.14718/) (2023)
- [On Training Implicit Meta-Learning With Applications to Inductive Weighing in Consistency Regularization](https://mlsanity.com/paper/2310.18741/) (2023)

## Abstract (arXiv)

This paper introduces SelfMatch, a semi-supervised learning method that combines the power of contrastive self-supervised learning and consistency regularization. SelfMatch consists of two stages: (1) self-supervised pre-training based on contrastive learning and (2) semi-supervised fine-tuning based on augmentation consistency regularization. We empirically demonstrate that SelfMatch achieves the state-of-the-art results on standard benchmark datasets such as CIFAR-10 and SVHN. For example, for CIFAR-10 with 40 labeled examples, SelfMatch achieves 93.19% accuracy that outperforms the strong previous methods such as MixMatch (52.46%), UDA (70.95%), ReMixMatch (80.9%), and FixMatch (86.19%). We note that SelfMatch can close the gap between supervised learning (95.87%) and semi-supervised learning (93.19%) by using only a few labels for each class.