mlsanityData to 7 Oct 2026

Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment

arXiv 2301.10625 ↗First posted 25 Jan 2023cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

Active Learning (AL) aims to reduce the labeling burden by interactively selecting the most informative samples from a pool of unlabeled data. While there has been extensive research on improving AL query methods in recent years, some studies have questioned the effectiveness of AL compared to emerging paradigms such as semi-supervised (Semi-SL) and self-supervised learning (Self-SL), or a simple optimization of classifier configurations. Thus, today's AL literature presents an inconsistent and contradictory landscape, leaving practitioners uncertain about whether and how to use AL in their tasks. In this work, we make the case that this inconsistency arises from a lack of systematic and realistic evaluation of AL methods. Specifically, we identify five key pitfalls in the current literature that reflect the delicate considerations required for AL evaluation. Further, we present an evaluation framework that overcomes these pitfalls and thus enables meaningful statements about the performance of AL methods. To demonstrate the relevance of our protocol, we present a large-scale empirical study and benchmark for image classification spanning various data sets, query methods, AL settings, and training paradigms. Our findings clarify the inconsistent picture in the literature and enable us to give hands-on recommendations for practitioners. The benchmark is hosted at https://github.com/IML-DKFZ/realistic-al .

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 25 Jan 2023.

Builds on

37 of 60 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  9. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  10. 2017Deep Bayesian Active Learning with Image Datacited by 812
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2020Self-supervised Learning: Generative or Contrastivecited by 693
25 more
  1. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  2. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  3. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  4. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  5. 2019Learning Loss for Active Learningcited by 335
  6. 2019Variational Adversarial Active Learningcited by 284
  7. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 224
  8. 2020Contextual Diversity for Active Learningcited by 139
  9. 2021Batch Active Learning at Scalecited by 127
  10. 2019Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Costcited by 114
  11. 2020Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learningcited by 105
  12. 2022A Comparative Survey of Deep Active Learningcited by 88
  13. 2020Task-Aware Variational Adversarial Active Learningcited by 84
  14. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  15. 2019Bayesian Batch Active Learning as Sparse Subset Approximationcited by 68
  16. 2020Towards Robust and Reproducible Active Learning Using Neural Networkscited by 53
  17. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  18. 2021Reducing Label Effort: Self-Supervised meets Active Learningcited by 42
  19. 2019Parting with Illusions about Deep Active Learningcited by 38
  20. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  21. 2022PT4AL: Using Self-Supervised Pretext Tasks for Active Learningcited by 27
  22. 2020On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?cited by 18
  23. 2020Bayesian active learning for production, a systematic study and a reusable librarycited by 15
  24. 2021A Practical & Unified Notation for Information-Theoretic Quantities in MLcited by 4
  25. 2022Robust Deep Semi-Supervised Learning: A Brief Introductioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 8; 2026: 7.

25 papers.

Sort:

By area:

  1. 2023A comprehensive survey on deep active learning in medical image analysiscited by 32
  2. 2024ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentationcited by 25
  3. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  4. 2024Revisiting Active Learning in the Era of Vision Foundation Modelscited by 13
  5. 2024A Cross-Domain Benchmark for Active Learningcited by 11
  6. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  7. 2024A Unified Approach Towards Active Learning and Out-of-Distribution Detectioncited by 7
  8. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7
  9. 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
  10. 2024On the Fragility of Active Learners for Text Classificationcited by 2
  11. 2025Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learningcited by 2
  12. 2024Leveraging image captions for selective whole slide image annotationcited by 1
13 more
  1. 2025No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Successcited by 1
  2. 2025Decomposition Sampling for Efficient Region Annotations in Active Learningcited by 1
  3. 2026Finally Outshining the Random Baseline: A Simple and Effective Solution for Active Learning in 3D Biomedical Imagingcited by 1
  4. 2026The Speed-up Factor: A Quantitative Multi-Iteration Active Learning Performance Metriccited by 1
  5. 2026BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learningcited by 1
  6. 2026Does Your Wildfire Prediction Model Actually Work, or Just Score Well?cited by 1
  7. 2026BioDCASE: Active Learning for Bioacousticscited by 1
  8. 2025Bayesian Active Learning By Distribution Disagreementcited by 0
  9. 2025The Role of Active Learning in Modern Machine Learningcited by 0
  10. 2025Calibrated Uncertainty Sampling for Active Learningcited by 0
  11. 2025Active Learning with Task-Driven Representations for Messy Poolscited by 0
  12. 2026Adaptive Active Learning for Regression via Reinforcement Learningcited by 0
  13. 2026QueryMarket: Cost-Aware Online Active Learning in Data Marketscited by 0