mlsanityData to 14 Oct 2026

An Expanded Benchmark that Rediscovers and Affirms the Edge of Uncertainty Sampling for Active Learning in Tabular Datasets

arXiv 2306.08954 ↗First posted 15 Jun 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Active Learning (AL) addresses the crucial challenge of enabling machines to efficiently gather labeled examples through strategic queries. Among the many AL strategies, Uncertainty Sampling (US) stands out as one of the most widely adopted. US queries the example(s) that the current model finds uncertain, proving to be both straightforward and effective. Despite claims in the literature suggesting superior alternatives to US, community-wide acceptance remains elusive. In fact, existing benchmarks for tabular datasets present conflicting conclusions on the continued competitiveness of US. In this study, we review the literature on AL strategies in the last decade and build the most comprehensive open-source AL benchmark to date to understand the relative merits of different AL strategies. The benchmark surpasses existing ones by encompassing a broader coverage of strategies, models, and data. Through our investigation of the conflicting conclusions in existing tabular AL benchmarks by evaluation under broad AL experimental settings, we uncover fresh insights into the often-overlooked issue of using machine learning models--**model compatibility** in the context of US. Specifically, we notice that adopting the different models for the querying unlabeled examples and learning tasks would degrade US's effectiveness. Notably, our findings affirm that US maintains a competitive edge over other strategies when paired with compatible models. These findings have practical implications and provide a concrete recipe for AL practitioners, empowering them to make informed decisions when working with tabular classifications with limited labeled data. The code for this project is available on https://github.com/ariapoy/active-learning-benchmark.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 85 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

33 of 85 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2017Deep Bayesian Active Learning with Image Datacited by 819
  5. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  6. 2019Learning Loss for Active Learningcited by 336
  7. 2019Variational Adversarial Active Learningcited by 284
  8. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  9. 2017Learning Active Learning from Datacited by 101
  10. 2019Deep Active Learning: Unified and Principled Method for Query and Trainingcited by 93
  11. 2022A Comparative Survey of Deep Active Learningcited by 89
  12. 2021Multiple instance active learning for object detectioncited by 77
21 more
  1. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  2. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
  3. 2020ASGN: An Active Semi-supervised Graph Neural Network for Molecular Property Predictioncited by 58
  4. 2020Towards Robust and Reproducible Active Learning Using Neural Networkscited by 54
  5. 2018Active Learning for Deep Object Detectioncited by 50
  6. 2021Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentationcited by 45
  7. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  8. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  9. 2016A Benchmark and Comparison of Active Learning for Logistic Regressioncited by 36
  10. 2018modAL: A modular active learning framework for Pythoncited by 35
  11. 2020Fair Active Learningcited by 30
  12. 2018Active Learning for Regression Using Greedy Samplingcited by 26
  13. 2017libact: Pool-based Active Learning in Pythoncited by 12
  14. 2018On the Relationship between Data Efficiency and Error for Uncertainty Samplingcited by 12
  15. 2019ALiPy: Active Learning in Pythoncited by 11
  16. 2022Is margin all you need? An extensive empirical study of active learning on tabular datacited by 10
  17. 2022DECAL: DEployable Clinical Active Learningcited by 7
  18. 2021Deep Active Learning for Text Classification with Diverse Interpretationscited by 5
  19. 2020A Graph-Based Approach for Active Learning in Regressioncited by 4
  20. 2020Learning active learning at the crossroads? evaluation and discussioncited by 3
  21. 2019Privacy-preserving Active Learning on Sensitive Data for User Intent Classificationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024ALPBench: A Benchmark for Active Learning Pipelines on Tabular Datacited by 5
  2. 2023On the Query Strategies for Efficient Online Active Distillationcited by 1
  3. 2024regAL: Python Package for Active Learning of Regression Problemscited by 0
  4. 2025Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Gridcited by 0