# Reducing Label Effort: Self-Supervised meets Active Learning

arXiv: https://arxiv.org/abs/2108.11458 · HTML: https://mlsanity.com/paper/2108.11458/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 25 Aug 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [What makes instance discrimination good for transfer learning?](https://mlsanity.com/paper/2006.06606/) (2020)
- [Task-Aware Variational Adversarial Active Learning](https://mlsanity.com/paper/2002.04709/) (2020)
- [State-Relabeling Adversarial Active Learning](https://mlsanity.com/paper/2004.04943/) (2020)
- [Exploiting Unlabeled Data in CNNs by Self-supervised Learning to Rank](https://mlsanity.com/paper/1902.06285/) (2019)
- [VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning](https://mlsanity.com/paper/2003.11249/) (2020)
- [Temporal Coherence for Active Learning in Videos](https://mlsanity.com/paper/1908.11757/) (2019)
- [Learning to Rank for Active Learning: A Listwise Approach](https://mlsanity.com/paper/2008.00078/) (2020)
- [A Variance Maximization Criterion for Active Learning](https://mlsanity.com/paper/1706.07642/) (2017)
- [When Deep Learners Change Their Mind: Learning Dynamics for Active Learning](https://mlsanity.com/paper/2107.14707/) (2021)
- [Active Transfer Learning with Zero-Shot Priors: Reusing Past Datasets for Future Tasks](https://mlsanity.com/paper/1510.01544/) (2015)
- [Information-Theoretic Active Learning for Content-Based Image Retrieval](https://mlsanity.com/paper/1809.02337/) (2018)

## Built on by (most cited first, 30 of 42)

- [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://mlsanity.com/paper/2202.02794/) (2022)
- [ActiveNeRF: Learning where to See with Uncertainty Estimation](https://mlsanity.com/paper/2209.08546/) (2022)
- [Active Learning Through a Covering Lens](https://mlsanity.com/paper/2205.11320/) (2022)
- [Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm](https://mlsanity.com/paper/2303.14382/) (2023)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [PT4AL: Using Self-Supervised Pretext Tasks for Active Learning](https://mlsanity.com/paper/2201.07459/) (2022)
- [Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment](https://mlsanity.com/paper/2301.10625/) (2023)
- [Making Your First Choice: To Address Cold Start Problem in Vision Active Learning](https://mlsanity.com/paper/2210.02442/) (2022)
- [Warm Start Active Learning with Proxy Labels & Selection via Semi-Supervised Fine-Tuning](https://mlsanity.com/paper/2209.06285/) (2022)
- [Active learning for medical image segmentation with stochastic batches](https://mlsanity.com/paper/2301.07670/) (2023)
- [Class-Balanced Active Learning for Image Classification](https://mlsanity.com/paper/2110.04543/) (2021)
- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)
- [Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation](https://mlsanity.com/paper/2207.10942/) (2022)
- [Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation](https://mlsanity.com/paper/2111.10932/) (2021)
- [Deep Active Ensemble Sampling For Image Classification](https://mlsanity.com/paper/2210.05770/) (2022)
- [Deep Active Learning for Computer Vision: Past and Future](https://mlsanity.com/paper/2211.14819/) (2022)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models](https://mlsanity.com/paper/2403.01101/) (2024)
- [Rethinking the Data Annotation Process for Multi-view 3D Pose Estimation with Active Learning and Self-Training](https://mlsanity.com/paper/2112.13709/) (2021)
- [MoBYv2AL: Self-supervised Active Learning for Image Classification](https://mlsanity.com/paper/2301.01531/) (2023)
- [Distributed Traffic Synthesis and Classification in Edge Networks: A Federated Self-supervised Learning Approach](https://mlsanity.com/paper/2302.00207/) (2023)
- [NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage](https://mlsanity.com/paper/2306.04099/) (2023)
- [ActiveDC: Distribution Calibration for Active Finetuning](https://mlsanity.com/paper/2311.07634/) (2023)
- [Augmentation-aware Self-supervised Learning with Conditioned Projector](https://mlsanity.com/paper/2306.06082/) (2023)
- [How To Overcome Confirmation Bias in Semi-Supervised Image Classification By Active Learning](https://mlsanity.com/paper/2308.08224/) (2023)
- [VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness](https://mlsanity.com/paper/2401.07853/) (2024)
- [Mixtraining: A Better Trade-Off Between Compute and Performance](https://mlsanity.com/paper/2502.19513/) (2025)
- [AutoLTS: Automating Cycling Stress Assessment via Contrastive Learning and Spatial Post-processing](https://mlsanity.com/paper/2308.07580/) (2023)
- [Improving Online Source-free Domain Adaptation for Object Detection by Unsupervised Data Acquisition](https://mlsanity.com/paper/2310.19258/) (2023)
- [Active Transfer Prototypical Network: An Efficient Labeling Algorithm for Time-Series Data](https://mlsanity.com/paper/2209.14199/) (2022)

## Abstract (arXiv)

Active learning is a paradigm aimed at reducing the annotation effort by training the model on actively selected informative and/or representative samples. Another paradigm to reduce the annotation effort is self-training that learns from a large amount of unlabeled data in an unsupervised way and fine-tunes on few labeled samples. Recent developments in self-training have achieved very impressive results rivaling supervised learning on some datasets. The current work focuses on whether the two paradigms can benefit from each other. We studied object recognition datasets including CIFAR10, CIFAR100 and Tiny ImageNet with several labeling budgets for the evaluations. Our experiments reveal that self-training is remarkably more efficient than active learning at reducing the labeling effort, that for a low labeling budget, active learning offers no benefit to self-training, and finally that the combination of active learning and self-training is fruitful when the labeling budget is high. The performance gap between active learning trained either with self-training or from scratch diminishes as we approach to the point where almost half of the dataset is labeled.