# Dynamic Ensemble Active Learning: A Non-Stationary Bandit with Expert Advice

arXiv: https://arxiv.org/abs/1810.07778 · HTML: https://mlsanity.com/paper/1810.07778/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2018.

## Built on by (most cited first, 7 of 7)

- [Algorithm Selection for Deep Active Learning with Imbalanced Datasets](https://mlsanity.com/paper/2302.07317/) (2023)
- [How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget](https://mlsanity.com/paper/2306.03543/) (2023)
- [Onception: Active Learning with Expert Advice for Real World Machine Translation](https://mlsanity.com/paper/2203.04507/) (2022)
- [Mitigating Biases in Collective Decision-Making: Enhancing Performance in the Face of Fake News](https://mlsanity.com/paper/2403.08829/) (2024)
- [AutoAL: Automated Active Learning with Differentiable Query Strategy Search](https://mlsanity.com/paper/2410.13853/) (2024)
- [PB-OEL: A Performance-Bounded Online Ensemble Learning Framework With Mixed Feedback for Real-Time Safety Assessment](https://mlsanity.com/paper/2503.15581/) (2025)
- [Soft Strategy Selection for Batch-Mode Active Learning](https://mlsanity.com/paper/2610.05544/) (2026)

## Abstract (arXiv)

Active learning aims to reduce annotation cost by predicting which samples are useful for a human teacher to label. However it has become clear there is no best active learning algorithm. Inspired by various philosophies about what constitutes a good criteria, different algorithms perform well on different datasets. This has motivated research into ensembles of active learners that learn what constitutes a good criteria in a given scenario, typically via multi-armed bandit algorithms. Though algorithm ensembles can lead to better results, they overlook the fact that not only does algorithm efficacy vary across datasets, but also during a single active learning session. That is, the best criteria is non-stationary. This breaks existing algorithms' guarantees and hampers their performance in practice. In this paper, we propose dynamic ensemble active learning as a more general and promising research direction. We develop a dynamic ensemble active learner based on a non-stationary multi-armed bandit with expert advice algorithm. Our dynamic ensemble selects the right criteria at each step of active learning. It has theoretical guarantees, and shows encouraging results on 13 popular datasets.