# Does Deep Active Learning Work in the Wild?

arXiv: https://arxiv.org/abs/2302.00098 · HTML: https://mlsanity.com/paper/2302.00098/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 16 of 16)

- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Batch Active Learning at Scale](https://mlsanity.com/paper/2107.14263/) (2021)
- [Less is more: sampling chemical space with active learning](https://mlsanity.com/paper/1801.09319/) (2018)
- [A Framework and Benchmark for Deep Batch Active Learning for Regression](https://mlsanity.com/paper/2203.09410/) (2022)
- [Effective Evaluation of Deep Active Learning on Image Classification Tasks](https://mlsanity.com/paper/2106.15324/) (2021)
- [A Benchmark and Comparison of Active Learning for Logistic Regression](https://mlsanity.com/paper/1611.08618/) (2016)
- [Pool-Based Sequential Active Learning for Regression](https://mlsanity.com/paper/1805.04735/) (2018)
- [Benchmarking deep inverse models over time, and the neural-adjoint method](https://mlsanity.com/paper/2009.12919/) (2020)
- [In Defense of Core-set: A Density-aware Core-set Selection for Active Learning](https://mlsanity.com/paper/2206.04838/) (2022)
- [Active Learning for Regression Using Greedy Sampling](https://mlsanity.com/paper/1808.04245/) (2018)
- [Dropout-based Active Learning for Regression](https://mlsanity.com/paper/1806.09856/) (2018)
- [Active and Continuous Exploration with Deep Neural Networks and Expected Model Output Changes](https://mlsanity.com/paper/1612.06129/) (2016)
- [A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching](https://mlsanity.com/paper/2003.13114/) (2020)
- [Unsupervised Pool-Based Active Learning for Linear Regression](https://mlsanity.com/paper/2001.05028/) (2020)

## Built on by (most cited first, 1 of 1)

- [PINNACLE: PINN Adaptive ColLocation and Experimental points selection](https://mlsanity.com/paper/2404.07662/) (2024)

## Abstract (arXiv)

Deep active learning (DAL) methods have shown significant improvements in sample efficiency compared to simple random sampling. While these studies are valuable, they nearly always assume that optimal DAL hyperparameter (HP) settings are known in advance, or optimize the HPs through repeating DAL several times with different HP settings. Here, we argue that in real-world settings, or in the wild, there is significant uncertainty regarding good HPs, and their optimization contradicts the premise of using DAL (i.e., we require labeling efficiency). In this study, we evaluate the performance of eleven modern DAL methods on eight benchmark problems as we vary a key HP shared by all methods: the pool ratio. Despite adjusting only one HP, our results indicate that eight of the eleven DAL methods sometimes underperform relative to simple random sampling and some frequently perform worse. Only three methods always outperform random sampling (albeit narrowly), and we find that these methods all utilize diversity to select samples - a relatively simple criterion. Our findings reveal the limitations of existing DAL methods when deployed in the wild, and present this as an important new open problem in the field.