# Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler

arXiv: https://arxiv.org/abs/2211.02233 · HTML: https://mlsanity.com/paper/2211.02233/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [More Adaptive Algorithms for Adversarial Bandits](https://mlsanity.com/paper/1801.03265/) (2018)
- [Diverse mini-batch Active Learning](https://mlsanity.com/paper/1901.05954/) (2019)
- [Crowdsourcing in Computer Vision](https://mlsanity.com/paper/1611.02145/) (2016)
- [Active Learning from Weak and Strong Labelers](https://mlsanity.com/paper/1510.02847/) (2015)
- [Taking a hint: How to leverage loss predictors in contextual bandits?](https://mlsanity.com/paper/2003.01922/) (2020)
- [Efficient and Parsimonious Agnostic Active Learning](https://mlsanity.com/paper/1506.08669/) (2015)
- [Active Learning from Imperfect Labelers](https://mlsanity.com/paper/1610.09730/) (2016)
- [On Unifying Misinformation Detection](https://mlsanity.com/paper/2104.05243/) (2021)
- [Top K Ranking for Multi-Armed Bandit with Noisy Evaluations](https://mlsanity.com/paper/2112.06517/) (2021)

## Built on by (most cited first, 4 of 4)

- [Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separation](https://mlsanity.com/paper/2312.09196/) (2023)
- [Active Learning with a Noisy Annotator](https://mlsanity.com/paper/2504.04506/) (2025)
- [An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations](https://mlsanity.com/paper/2604.23290/) (2026)
- [Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance](https://mlsanity.com/paper/2606.07630/) (2026)

## Abstract (arXiv)

Active learning with strong and weak labelers considers a practical setting where we have access to both costly but accurate strong labelers and inaccurate but cheap predictions provided by weak labelers. We study this problem in the streaming setting, where decisions must be taken textitonline. We design a novel algorithmic template, Weak Labeler Active Cover (WL-AC), that is able to robustly leverage the lower quality weak labelers to reduce the query complexity while retaining the desired level of accuracy. Prior active learning algorithms with access to weak labelers learn a difference classifier which predicts where the weak labels differ from strong labelers; this requires the strong assumption of realizability of the difference classifier (Zhang and Chaudhuri,2015). WL-AC bypasses this textitrealizability assumption and thus is applicable to many real-world scenarios such as random corrupted weak labels and high dimensional family of difference classifiers (textite.g., deep neural nets). Moreover, WL-AC cleverly trades off evaluating the quality with full exploitation of weak labelers, which allows to convert any active learning strategy to one that can leverage weak labelers. We provide an instantiation of this template that achieves the optimal query complexity for any given weak labeler, without knowing its accuracy a-priori. Empirically, we propose an instantiation of the WL-AC template that can be efficiently implemented for large-scale models (textite.g., deep neural nets) and show its effectiveness on the corrupted-MNIST dataset by significantly reducing the number of labels while keeping the same accuracy as in passive learning.