# Cost-Accuracy Aware Adaptive Labeling for Active Learning

arXiv: https://arxiv.org/abs/2105.11418 · HTML: https://mlsanity.com/paper/2105.11418/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 35 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 2 of 2)

- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Learning From Noisy Singly-labeled Data](https://mlsanity.com/paper/1712.04577/) (2017)

## Built on by (most cited first, 13 of 13)

- [Diversity Enhanced Active Learning with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2110.14171/) (2021)
- [Algorithmic Fairness in Business Analytics: Directions for Research and Practice](https://mlsanity.com/paper/2207.10991/) (2022)
- [Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal](https://mlsanity.com/paper/2303.01560/) (2023)
- [Learning Complementary Policies for Human-AI Teams](https://mlsanity.com/paper/2302.02944/) (2023)
- [Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning](https://mlsanity.com/paper/2312.10116/) (2023)
- [Revisiting Active Learning under (Human) Label Variation](https://mlsanity.com/paper/2507.02593/) (2025)
- [More Data Can Lead Us Astray: Active Data Acquisition in the Presence of Label Bias](https://mlsanity.com/paper/2207.07723/) (2022)
- [Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains](https://mlsanity.com/paper/2508.17202/) (2025)
- [A Machine Learning Framework Towards Transparency in Experts' Decision Quality](https://mlsanity.com/paper/2110.11425/) (2021)
- [Active Learning with Weak Supervision for Gaussian Processes](https://mlsanity.com/paper/2204.08335/) (2022)
- [Learning to Project for Cross-Task Knowledge Distillation](https://mlsanity.com/paper/2403.14494/) (2024)
- [ADs: Active Data-sharing for Data Quality Assurance in Advanced Manufacturing Systems](https://mlsanity.com/paper/2404.00572/) (2024)
- [ALIVE: Warnings Before Exclusion in Budgeted Multi-Source Learning](https://mlsanity.com/paper/2607.29400/) (2026)

## Abstract (arXiv)

Conventional active learning algorithms assume a single labeler that produces noiseless label at a given, fixed cost, and aim to achieve the best generalization performance for given classifier under a budget constraint. However, in many real settings, different labelers have different labeling costs and can yield different labeling accuracies. Moreover, a given labeler may exhibit different labeling accuracies for different instances. This setting can be referred to as active learning with diverse labelers with varying costs and accuracies, and it arises in many important real settings. It is therefore beneficial to understand how to effectively trade-off between labeling accuracy for different instances, labeling costs, as well as the informativeness of training instances, so as to achieve the best generalization performance at the lowest labeling cost. In this paper, we propose a new algorithm for selecting instances, labelers (and their corresponding costs and labeling accuracies), that employs generalization bound of learning with label noise to select informative instances and labelers so as to achieve higher generalization accuracy at a lower cost. Our proposed algorithm demonstrates state-of-the-art performance on five UCI and a real crowdsourcing dataset.