# Active Learning from Weak and Strong Labelers

arXiv: https://arxiv.org/abs/1510.02847 · HTML: https://mlsanity.com/paper/1510.02847/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2015.

## Built on by (most cited first, 22 of 22)

- [Sample Efficient Learning of Predictors that Complement Humans](https://mlsanity.com/paper/2207.09584/) (2022)
- [RIM: Reliable Influence-based Active Learning on Graphs](https://mlsanity.com/paper/2110.14854/) (2021)
- [Active Imitation Learning with Noisy Guidance](https://mlsanity.com/paper/2005.12801/) (2020)
- [Multiple-Prediction-Powered Inference](https://mlsanity.com/paper/2603.27414/) (2026)
- [Annotation Curricula to Implicitly Train Non-Expert Annotators](https://mlsanity.com/paper/2106.02382/) (2021)
- [Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separation](https://mlsanity.com/paper/2312.09196/) (2023)
- [Probably Approximately Correct Labels](https://mlsanity.com/paper/2506.10908/) (2025)
- [Agnostic Multi-Group Active Learning](https://mlsanity.com/paper/2306.01922/) (2023)
- [Deep Active Learning in the Presence of Label Noise: A Survey](https://mlsanity.com/paper/2302.11075/) (2023)
- [Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler](https://mlsanity.com/paper/2211.02233/) (2022)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [Revisiting Active Learning under (Human) Label Variation](https://mlsanity.com/paper/2507.02593/) (2025)
- [Active Cost-aware Labeling of Streaming Data](https://mlsanity.com/paper/2304.06808/) (2023)
- [Accelerating Matroid Optimization through Fast Imprecise Oracles](https://mlsanity.com/paper/2402.02774/) (2024)
- [Active Learning with a Noisy Annotator](https://mlsanity.com/paper/2504.04506/) (2025)
- [FishNet: Deep Neural Networks for Low-Cost Fish Stock Estimation](https://mlsanity.com/paper/2403.10916/) (2024)
- [Interactive Machine Teaching by Labeling Rules and Instances](https://mlsanity.com/paper/2409.05199/) (2024)
- [Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection](https://mlsanity.com/paper/2512.12870/) (2025)
- [Top-k on a Budget: Adaptive Ranking with Weak and Strong Oracles](https://mlsanity.com/paper/2601.20989/) (2026)
- [An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations](https://mlsanity.com/paper/2604.23290/) (2026)
- [Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance](https://mlsanity.com/paper/2606.07630/) (2026)
- [MICRO: Multi-Fidelity Active Search for Severe Error Discovery](https://mlsanity.com/paper/2609.26025/) (2026)

## Abstract (arXiv)

An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses active learning with labels obtained from strong and weak labelers, where in addition to the standard active learning setting, we have an extra weak labeler which may occasionally provide incorrect labels. An example is learning to classify medical images where either expensive labels may be obtained from a physician (oracle or strong labeler), or cheaper but occasionally incorrect labels may be obtained from a medical resident (weak labeler). Our goal is to learn a classifier with low error on data labeled by the oracle, while using the weak labeler to reduce the number of label queries made to this labeler. We provide an active learning algorithm for this setting, establish its statistical consistency, and analyze its label complexity to characterize when it can provide label savings over using the strong labeler alone.