# On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?

arXiv: https://arxiv.org/abs/2011.08121 · HTML: https://mlsanity.com/paper/2011.08121/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 18 of 18)

- [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://mlsanity.com/paper/2202.02794/) (2022)
- [A Comparative Survey of Deep Active Learning](https://mlsanity.com/paper/2203.13450/) (2022)
- [Active Learning Through a Covering Lens](https://mlsanity.com/paper/2205.11320/) (2022)
- [Active Learning at the ImageNet Scale](https://mlsanity.com/paper/2111.12880/) (2021)
- [Unsupervised Selective Labeling for More Effective Semi-Supervised Learning](https://mlsanity.com/paper/2110.03006/) (2021)
- [Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment](https://mlsanity.com/paper/2301.10625/) (2023)
- [LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning](https://mlsanity.com/paper/2306.09910/) (2023)
- [How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget](https://mlsanity.com/paper/2306.03543/) (2023)
- [Revisiting Active Learning in the Era of Vision Foundation Models](https://mlsanity.com/paper/2401.14555/) (2024)
- [Making Better Use of Unlabelled Data in Bayesian Active Learning](https://mlsanity.com/paper/2404.17249/) (2024)
- [Deep Active Learning for Computer Vision: Past and Future](https://mlsanity.com/paper/2211.14819/) (2022)
- [An Empirical Study on the Efficacy of Deep Active Learning for Image Classification](https://mlsanity.com/paper/2212.03088/) (2022)
- [Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models](https://mlsanity.com/paper/2403.01101/) (2024)
- [NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage](https://mlsanity.com/paper/2306.04099/) (2023)
- [How To Overcome Confirmation Bias in Semi-Supervised Image Classification By Active Learning](https://mlsanity.com/paper/2308.08224/) (2023)
- [Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection](https://mlsanity.com/paper/2409.11653/) (2024)
- [Cold PAWS: Unsupervised class discovery and addressing the cold-start problem for semi-supervised learning](https://mlsanity.com/paper/2305.10071/) (2023)
- [Active Learning with Task-Driven Representations for Messy Pools](https://mlsanity.com/paper/2510.25926/) (2025)

## Abstract (arXiv)

Active learning is the set of techniques for intelligently labeling large unlabeled datasets to reduce the labeling effort. In parallel, recent developments in self-supervised and semi-supervised learning (S4L) provide powerful techniques, based on data-augmentation, contrastive learning, and self-training, that enable superior utilization of unlabeled data which led to a significant reduction in required labeling in the standard machine learning benchmarks. A natural question is whether these paradigms can be unified to obtain superior results. To this aim, this paper provides a novel algorithmic framework integrating self-supervised pretraining, active learning, and consistency-regularized self-training. We conduct extensive experiments with our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to isolate and assess the benefits of individual components which are evaluated using state-of-the-art methods (e.g.~Core-Set, VAAL, simCLR, FixMatch). Our experiments reveal two key insights: (i) Self-supervised pre-training significantly improves semi-supervised learning, especially in the few-label regime, (ii) The benefit of active learning is undermined and subsumed by S4L techniques. Specifically, we fail to observe any additional benefit of state-of-the-art active learning algorithms when combined with state-of-the-art S4L techniques.