mlsanityData to 14 Oct 2026

Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Grid

arXiv 2506.03817 ↗First posted 4 Jun 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Annotating data is a time-consuming and costly task, but it is inherently required for supervised machine learning. Active Learning (AL) is an established method that minimizes human labeling effort by iteratively selecting the most informative unlabeled samples for expert annotation, thereby improving the overall classification performance. Even though AL has been known for decades, AL is still rarely used in real-world applications. As indicated in the two community web surveys among the NLP community about AL, two main reasons continue to hold practitioners back from using AL: first, the complexity of setting AL up, and second, a lack of trust in its effectiveness. We hypothesize that both reasons share the same culprit: the large hyperparameter space of AL. This mostly unexplored hyperparameter space often leads to misleading and irreproducible AL experiment results. In this study, we first compiled a large hyperparameter grid of over 4.6 million hyperparameter combinations, second, recorded the performance of all combinations in the so-far biggest conducted AL study, and third, analyzed the impact of each hyperparameter in the experiment results. In the end, we give recommendations about the influence of each hyperparameter, demonstrate the surprising influence of the concrete AL strategy implementation, and outline an experimental study design for reproducible AL experiments with minimal computational effort, thus contributing to more reproducible and trustworthy AL research in the future.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 90 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

32 of 90 references.

References · 32

  1. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  2. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  3. 2017OpenML Benchmarking Suitescited by 178
  4. 2021Active Learning by Acquiring Contrastive Examplescited by 156
  5. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  6. 2017Learning Active Learning from Datacited by 101
  7. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  8. 2022A Comparative Survey of Deep Active Learningcited by 89
  9. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  10. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
  11. 2020A Survey of Active Learning for Text Classification using Deep Neural Networkscited by 50
  12. 2021Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentationcited by 45
20 more
  1. 2016A Benchmark and Comparison of Active Learning for Logistic Regressioncited by 36
  2. 2021Cartography Active Learningcited by 26
  3. 2021Small-Text: Active Learning for Text Classification in Pythoncited by 19
  4. 2020On Initial Pools for Deep Active Learningcited by 18
  5. 2022DeepCAVE: An Interactive Analysis Tool for Automated Machine Learningcited by 14
  6. 2017libact: Pool-based Active Learning in Pythoncited by 12
  7. 2018Discovering General-Purpose Active Learning Strategiescited by 11
  8. 2019ALiPy: Active Learning in Pythoncited by 11
  9. 2022Is margin all you need? An extensive empirical study of active learning on tabular datacited by 10
  10. 2021Hitting the Target: Stopping Active Learning at the Cost-Based Optimumcited by 9
  11. 2022DECAL: DEployable Clinical Active Learningcited by 7
  12. 2024ALPBench: A Benchmark for Active Learning Pipelines on Tabular Datacited by 5
  13. 2020Rebuilding Trust in Active Learning with Actionable Metricscited by 4
  14. 2021Sample Noise Impact on Active Learningcited by 4
  15. 2023An Expanded Benchmark that Rediscovers and Affirms the Edge of Uncertainty Sampling for Active Learning in Tabular Datasetscited by 4
  16. 2014When does Active Learning Work?cited by 3
  17. 2020Learning active learning at the crossroads? evaluation and discussioncited by 3
  18. 2024On the Fragility of Active Learners for Text Classificationcited by 2
  19. 2025Reassessing Active Learning Adoption in Contemporary NLP: A Community Surveycited by 2
  20. 2022ImitAL: Learned Active Learning Strategy on Synthetic Datacited by 1

Built on by

Not cited yet.