# Towards Comparable Active Learning

arXiv: https://arxiv.org/abs/2311.18356 · HTML: https://mlsanity.com/paper/2311.18356/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets](https://mlsanity.com/paper/2202.02794/) (2022)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [Learning Active Learning from Data](https://mlsanity.com/paper/1703.03365/) (2017)
- [Deep Active Learning: Unified and Principled Method for Query and Training](https://mlsanity.com/paper/1911.09162/) (2019)
- [Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2010.03546/) (2020)
- [Sequential Graph Convolutional Network for Active Learning](https://mlsanity.com/paper/2006.10219/) (2020)
- [Practical Obstacles to Deploying Active Learning](https://mlsanity.com/paper/1807.04801/) (2018)
- [Towards Robust and Reproducible Active Learning Using Neural Networks](https://mlsanity.com/paper/2002.09564/) (2020)
- [Adversarial Sampling for Active Learning](https://mlsanity.com/paper/1808.06671/) (2018)
- [News Category Dataset](https://mlsanity.com/paper/2209.11429/) (2022)
- [Effective Evaluation of Deep Active Learning on Image Classification Tasks](https://mlsanity.com/paper/2106.15324/) (2021)
- [Learning to Sample: an Active Learning Framework](https://mlsanity.com/paper/1909.03585/) (2019)
- [An Empirical Study on the Efficacy of Deep Active Learning for Image Classification](https://mlsanity.com/paper/2212.03088/) (2022)

## Abstract (arXiv)

Active Learning has received significant attention in the field of machine learning for its potential in selecting the most informative samples for labeling, thereby reducing data annotation costs. However, we show that the reported lifts in recent literature generalize poorly to other domains leading to an inconclusive landscape in Active Learning research. Furthermore, we highlight overlooked problems for reproducing AL experiments that can lead to unfair comparisons and increased variance in the results. This paper addresses these issues by providing an Active Learning framework for a fair comparison of algorithms across different tasks and domains, as well as a fast and performant oracle algorithm for evaluation. To the best of our knowledge, we propose the first AL benchmark that tests algorithms in 3 major domains: Tabular, Image, and Text. We report empirical results for 6 widely used algorithms on 7 real-world and 2 synthetic datasets and aggregate them into a domain-specific ranking of AL algorithms.