mlsanityData to 7 Oct 2026

A Cross-Domain Benchmark for Active Learning

arXiv 2408.00426 ↗First posted 1 Aug 2024cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose CDALBench, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, CDALBench can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

22 of 42 references.

References · 22

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2017Deep Bayesian Active Learning with Image Datacited by 812
  4. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  5. 2018Deep learning for time series classification: a reviewcited by 575
  6. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  7. 2019Learning Loss for Active Learningcited by 335
  8. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
  9. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 123
  10. 2022A Comparative Survey of Deep Active Learningcited by 88
  11. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  12. 2020Sequential Graph Convolutional Network for Active Learningcited by 69
10 more
  1. 2018Practical Obstacles to Deploying Active Learningcited by 68
  2. 2020Towards Robust and Reproducible Active Learning Using Neural Networkscited by 53
  3. 2022News Category Datasetcited by 42
  4. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  5. 2022GALAXY: Graph-based Active Learning at the Extremecited by 29
  6. 2023Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessmentcited by 25
  7. 2023LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learningcited by 20
  8. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  9. 2022Is margin all you need? An extensive empirical study of active learning on tabular datacited by 10
  10. 2022An Empirical Study on the Efficacy of Deep Active Learning for Image Classificationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2026Next Generation Active Learning: Mixture of LLMs in the Loopcited by 3
  2. 2025No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Successcited by 1
  3. 2025MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysiscited by 1
  4. 2025ALScope: A Unified Toolkit for Deep Active Learningcited by 1
  5. 2026BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learningcited by 1
  6. 2025Active Learning Methods for Efficient Data Utilization and Model Performance Enhancementcited by 0
  7. 2025The Role of Active Learning in Modern Machine Learningcited by 0
  8. 2025Calibrated Uncertainty Sampling for Active Learningcited by 0
  9. 2025Batch Bayesian Active Learning with Partial Batch Label Samplingcited by 0
  10. 2026Federated Active Learning Under Extreme Non-IID and Global Class Imbalancecited by 0
  11. 2026Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labelscited by 0