# Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Grid

arXiv: https://arxiv.org/abs/2506.03817 · HTML: https://mlsanity.com/paper/2506.03817/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 90 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [Cold-start Active Learning through Self-supervised Language Modeling](https://mlsanity.com/paper/2010.09535/) (2020)
- [Learning Active Learning from Data](https://mlsanity.com/paper/1703.03365/) (2017)
- [Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study](https://mlsanity.com/paper/1808.05697/) (2018)
- [A Comparative Survey of Deep Active Learning](https://mlsanity.com/paper/2203.13450/) (2022)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [A Survey of Active Learning for Text Classification using Deep Neural Networks](https://mlsanity.com/paper/2008.07267/) (2020)
- [Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentation](https://mlsanity.com/paper/2101.02323/) (2021)
- [A Benchmark and Comparison of Active Learning for Logistic Regression](https://mlsanity.com/paper/1611.08618/) (2016)
- [Cartography Active Learning](https://mlsanity.com/paper/2109.04282/) (2021)
- [Small-Text: Active Learning for Text Classification in Python](https://mlsanity.com/paper/2107.10314/) (2021)
- [On Initial Pools for Deep Active Learning](https://mlsanity.com/paper/2011.14696/) (2020)
- [DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning](https://mlsanity.com/paper/2206.03493/) (2022)
- [libact: Pool-based Active Learning in Python](https://mlsanity.com/paper/1710.00379/) (2017)
- [Discovering General-Purpose Active Learning Strategies](https://mlsanity.com/paper/1810.04114/) (2018)
- [ALiPy: Active Learning in Python](https://mlsanity.com/paper/1901.03802/) (2019)
- [Is margin all you need? An extensive empirical study of active learning on tabular data](https://mlsanity.com/paper/2210.03822/) (2022)
- [Hitting the Target: Stopping Active Learning at the Cost-Based Optimum](https://mlsanity.com/paper/2110.03802/) (2021)
- [DECAL: DEployable Clinical Active Learning](https://mlsanity.com/paper/2206.10120/) (2022)
- [ALPBench: A Benchmark for Active Learning Pipelines on Tabular Data](https://mlsanity.com/paper/2406.17322/) (2024)
- [Rebuilding Trust in Active Learning with Actionable Metrics](https://mlsanity.com/paper/2012.11365/) (2020)
- [Sample Noise Impact on Active Learning](https://mlsanity.com/paper/2109.01372/) (2021)
- [An Expanded Benchmark that Rediscovers and Affirms the Edge of Uncertainty Sampling for Active Learning in Tabular Datasets](https://mlsanity.com/paper/2306.08954/) (2023)
- [When does Active Learning Work?](https://mlsanity.com/paper/1408.1319/) (2014)
- [Learning active learning at the crossroads? evaluation and discussion](https://mlsanity.com/paper/2012.09631/) (2020)
- [On the Fragility of Active Learners for Text Classification](https://mlsanity.com/paper/2403.15744/) (2024)

## Abstract (arXiv)

Annotating data is a time-consuming and costly task, but it is inherently required for supervised machine learning. Active Learning (AL) is an established method that minimizes human labeling effort by iteratively selecting the most informative unlabeled samples for expert annotation, thereby improving the overall classification performance. Even though AL has been known for decades, AL is still rarely used in real-world applications. As indicated in the two community web surveys among the NLP community about AL, two main reasons continue to hold practitioners back from using AL: first, the complexity of setting AL up, and second, a lack of trust in its effectiveness. We hypothesize that both reasons share the same culprit: the large hyperparameter space of AL. This mostly unexplored hyperparameter space often leads to misleading and irreproducible AL experiment results. In this study, we first compiled a large hyperparameter grid of over 4.6 million hyperparameter combinations, second, recorded the performance of all combinations in the so-far biggest conducted AL study, and third, analyzed the impact of each hyperparameter in the experiment results. In the end, we give recommendations about the influence of each hyperparameter, demonstrate the surprising influence of the concrete AL strategy implementation, and outline an experimental study design for reproducible AL experiments with minimal computational effort, thus contributing to more reproducible and trustworthy AL research in the future.