# Effective Evaluation of Deep Active Learning on Image Classification Tasks

arXiv: https://arxiv.org/abs/2106.15324 · HTML: https://mlsanity.com/paper/2106.15324/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Adversarial Active Learning for Deep Networks: a Margin Based Approach](https://mlsanity.com/paper/1802.09841/) (2018)
- [Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning](https://mlsanity.com/paper/2010.05627/) (2020)
- [Submodular Combinatorial Information Measures with Applications in Machine Learning](https://mlsanity.com/paper/2006.15412/) (2020)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Vision](https://mlsanity.com/paper/1901.01151/) (2019)
- [Towards Robust and Reproducible Active Learning Using Neural Networks](https://mlsanity.com/paper/2002.09564/) (2020)
- [apricot: Submodular selection for data summarization in Python](https://mlsanity.com/paper/1906.03543/) (2019)

## Built on by (most cited first, 30 of 37)

- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Active Learning at the ImageNet Scale](https://mlsanity.com/paper/2111.12880/) (2021)
- [TALISMAN: Targeted Active Learning for Object Detection with Rare Classes and Slices using Submodular Mutual Information](https://mlsanity.com/paper/2112.00166/) (2021)
- [Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment](https://mlsanity.com/paper/2301.10625/) (2023)
- [LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning](https://mlsanity.com/paper/2306.09910/) (2023)
- [Algorithm Selection for Deep Active Learning with Imbalanced Datasets](https://mlsanity.com/paper/2302.07317/) (2023)
- [Deep Active Learning with Noise Stability](https://mlsanity.com/paper/2205.13340/) (2022)
- [ActiveGLAE: A Benchmark for Deep Active Learning with Transformers](https://mlsanity.com/paper/2306.10087/) (2023)
- [A Cross-Domain Benchmark for Active Learning](https://mlsanity.com/paper/2408.00426/) (2024)
- [Navya3DSeg -- Navya 3D Semantic Segmentation Dataset & split generation for autonomous vehicles](https://mlsanity.com/paper/2302.08292/) (2023)
- [Accelerating Batch Active Learning Using Continual Learning Techniques](https://mlsanity.com/paper/2305.06408/) (2023)
- [SCoRe: Submodular Combinatorial Representation Learning](https://mlsanity.com/paper/2310.00165/) (2023)
- [An Empirical Study on the Efficacy of Deep Active Learning for Image Classification](https://mlsanity.com/paper/2212.03088/) (2022)
- [Beyond Active Learning: Leveraging the Full Potential of Human Interaction via Auto-Labeling, Human Correction, and Human Verification](https://mlsanity.com/paper/2306.01277/) (2023)
- [An Expanded Benchmark that Rediscovers and Affirms the Edge of Uncertainty Sampling for Active Learning in Tabular Datasets](https://mlsanity.com/paper/2306.08954/) (2023)
- [Beyond Labeling Oracles: What does it mean to steal ML models?](https://mlsanity.com/paper/2310.01959/) (2023)
- [Feasibility Study on Active Learning of Smart Surrogates for Scientific Simulations](https://mlsanity.com/paper/2407.07674/) (2024)
- [Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection](https://mlsanity.com/paper/2510.00303/) (2025)
- [Learning to Rank for Active Learning via Multi-Task Bilevel Optimization](https://mlsanity.com/paper/2310.17044/) (2023)
- [LiDAR dataset distillation within bayesian active learning framework: Understanding the effect of data augmentation](https://mlsanity.com/paper/2202.02661/) (2022)
- [STREAMLINE: Streaming Active Learning for Realistic Multi-Distributional Settings](https://mlsanity.com/paper/2305.10643/) (2023)
- [How To Overcome Confirmation Bias in Semi-Supervised Image Classification By Active Learning](https://mlsanity.com/paper/2308.08224/) (2023)
- [An Interactive Visualization Tool for Understanding Active Learning](https://mlsanity.com/paper/2111.04936/) (2021)
- [ImitAL: Learned Active Learning Strategy on Synthetic Data](https://mlsanity.com/paper/2208.11636/) (2022)
- [Does Deep Active Learning Work in the Wild?](https://mlsanity.com/paper/2302.00098/) (2023)
- [Benchmarking of Query Strategies: Towards Future Deep Active Learning](https://mlsanity.com/paper/2312.05751/) (2023)
- [Finally Outshining the Random Baseline: A Simple and Effective Solution for Active Learning in 3D Biomedical Imaging](https://mlsanity.com/paper/2601.13677/) (2026)
- [SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition](https://mlsanity.com/paper/2602.05162/) (2026)
- [SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning](https://mlsanity.com/paper/2605.12872/) (2026)
- [Evaluating the effect of data augmentation and BALD heuristics on distillation of Semantic-KITTI dataset](https://mlsanity.com/paper/2302.10679/) (2023)

## Abstract (arXiv)

With the goal of making deep learning more label-efficient, a growing number of papers have been studying active learning (AL) for deep models. However, there are a number of issues in the prevalent experimental settings, mainly stemming from a lack of unified implementation and benchmarking. Issues in the current literature include sometimes contradictory observations on the performance of different AL algorithms, unintended exclusion of important generalization approaches such as data augmentation and SGD for optimization, a lack of study of evaluation facets like the labeling efficiency of AL, and little or no clarity on the scenarios in which AL outperforms random sampling (RS). In this work, we present a unified re-implementation of state-of-the-art AL algorithms in the context of image classification via our new open-source AL toolkit DISTIL, and we carefully study these issues as facets of effective evaluation. On the positive side, we show that AL techniques are 2× to 4× more label-efficient compared to RS with the use of data augmentation. Surprisingly, when data augmentation is included, there is no longer a consistent gain in using BADGE, a state-of-the-art approach, over simple uncertainty sampling. We then do a careful analysis of how existing approaches perform with varying amounts of redundancy and number of examples per class. Finally, we provide several insights for AL practitioners to consider in future work, such as the effect of the AL batch size, the effect of initialization, the importance of retraining the model at every round, and other insights.