mlsanityData to 7 Oct 2026

Towards Comparable Active Learning

arXiv 2311.18356 ↗First posted 30 Nov 2023cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Active Learning has received significant attention in the field of machine learning for its potential in selecting the most informative samples for labeling, thereby reducing data annotation costs. However, we show that the reported lifts in recent literature generalize poorly to other domains leading to an inconclusive landscape in Active Learning research. Furthermore, we highlight overlooked problems for reproducing AL experiments that can lead to unfair comparisons and increased variance in the results. This paper addresses these issues by providing an Active Learning framework for a fair comparison of algorithms across different tasks and domains, as well as a fast and performant oracle algorithm for evaluation. To the best of our knowledge, we propose the first AL benchmark that tests algorithms in 3 major domains: Tabular, Image, and Text. We report empirical results for 6 widely used algorithms on 7 real-world and 2 synthetic datasets and aggregate them into a domain-specific ranking of AL algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

19 of 30 references.

References · 19

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2017Deep Bayesian Active Learning with Image Datacited by 812
  4. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  5. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  6. 2019Learning Loss for Active Learningcited by 335
  7. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
  8. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 123
  9. 2017Learning Active Learning from Datacited by 100
  10. 2019Deep Active Learning: Unified and Principled Method for Query and Trainingcited by 92
  11. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  12. 2020Sequential Graph Convolutional Network for Active Learningcited by 69
7 more
  1. 2018Practical Obstacles to Deploying Active Learningcited by 68
  2. 2020Towards Robust and Reproducible Active Learning Using Neural Networkscited by 53
  3. 2018Adversarial Sampling for Active Learningcited by 51
  4. 2022News Category Datasetcited by 42
  5. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  6. 2019Learning to Sample: an Active Learning Frameworkcited by 11
  7. 2022An Empirical Study on the Efficacy of Deep Active Learning for Image Classificationcited by 6

Built on by

Not cited yet.