# A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification

arXiv: https://arxiv.org/abs/2109.11301 · HTML: https://mlsanity.com/paper/2109.11301/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 203 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 25 of 25)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Expectation Propagation for approximate Bayesian inference](https://mlsanity.com/paper/1301.2294/) (2013)
- [Deep Learning based Recommender System: A Survey and New Perspectives](https://mlsanity.com/paper/1707.07435/) (2017)
- [A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective](https://mlsanity.com/paper/1811.03402/) (2018)
- [Extreme clicking for efficient object annotation](https://mlsanity.com/paper/1708.02750/) (2017)
- [Algorithms for multi-armed bandit problems](https://mlsanity.com/paper/1402.6028/) (2014)
- [Active Learning for Cost-Sensitive Classification](https://mlsanity.com/paper/1703.01014/) (2017)
- [ALICE: Active Learning with Contrastive Natural Language Explanations](https://mlsanity.com/paper/2009.10259/) (2020)
- [Active classification with comparison queries](https://mlsanity.com/paper/1704.03564/) (2017)
- [Active Learning with Partial Feedback](https://mlsanity.com/paper/1802.07427/) (2018)
- [Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience](https://mlsanity.com/paper/2001.09219/) (2020)
- [Noise-tolerant, Reliable Active Classification with Comparison Queries](https://mlsanity.com/paper/2001.05497/) (2020)
- [Leveraging Crowdsourcing Data For Deep Active Learning - An Application: Learning Intents in Alexa](https://mlsanity.com/paper/1803.04223/) (2018)
- [Stopping Active Learning based on Predicted Change of F Measure for Text Classification](https://mlsanity.com/paper/1901.09118/) (2019)
- [A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control](https://mlsanity.com/paper/1812.02736/) (2018)
- [Limitations of Assessing Active Learning Performance at Runtime](https://mlsanity.com/paper/1901.10338/) (2019)
- [Toward Optimal Probabilistic Active Learning Using a Bayesian Approach](https://mlsanity.com/paper/2006.01732/) (2020)
- [Human-In-The-Loop Automatic Program Repair](https://mlsanity.com/paper/1912.07758/) (2019)
- [An Active Learning Approach for Jointly Estimating Worker Performance and Annotation Reliability with Crowdsourced Data](https://mlsanity.com/paper/1401.3836/) (2014)
- [Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks](https://mlsanity.com/paper/1905.04981/) (2019)
- [Semi-Supervised Active Learning for Support Vector Machines: A Novel Approach that Exploits Structure Information in Data](https://mlsanity.com/paper/1610.03995/) (2016)
- [Collaborative Interactive Learning -- A clarification of terms and a differentiation from other research fields](https://mlsanity.com/paper/1905.07264/) (2019)

## Built on by (most cited first, 17 of 17)

- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Improving Model Evaluation using SMART Filtering of Benchmark Datasets](https://mlsanity.com/paper/2410.20245/) (2024)
- [Controllable Textual Inversion for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2304.05265/) (2023)
- [ActiveGLAE: A Benchmark for Deep Active Learning with Transformers](https://mlsanity.com/paper/2306.10087/) (2023)
- [Multi-annotator Deep Learning: A Probabilistic Framework for Classification](https://mlsanity.com/paper/2304.02539/) (2023)
- [Learning To Guide Human Decision Makers With Vision-Language Models](https://mlsanity.com/paper/2403.16501/) (2024)
- [Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension](https://mlsanity.com/paper/2405.03386/) (2024)
- [dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans](https://mlsanity.com/paper/2407.20950/) (2024)
- [Enhancing Cost Efficiency in Active Learning with Candidate Set Query](https://mlsanity.com/paper/2502.06209/) (2025)
- [ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP](https://mlsanity.com/paper/2308.02537/) (2023)
- [Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks](https://mlsanity.com/paper/2407.03895/) (2024)
- [DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning](https://mlsanity.com/paper/2502.00305/) (2025)
- [Interactive Explainable Anomaly Detection for Industrial Settings](https://mlsanity.com/paper/2410.12817/) (2024)
- [Augmenting Image Annotation: A Human-LMM Collaborative Framework for Efficient Object Selection and Label Generation](https://mlsanity.com/paper/2503.11096/) (2025)
- [A Review of Uncertainty Calibration in Pretrained Object Detectors](https://mlsanity.com/paper/2210.02935/) (2022)
- [Active Label Refinement for Semantic Segmentation of Satellite Images](https://mlsanity.com/paper/2309.06159/) (2023)
- [crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels](https://mlsanity.com/paper/2504.09085/) (2025)

## Abstract (arXiv)

Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, a wide range of novel AL strategies has been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: (1) They explicitly consider (multiple) human annotators whose performances can be affected by various factors, such as missing expertise. (2) They generalize the interaction with human annotators by considering different query and annotation types, such as asking an annotator for feedback on an inferred classification rule. (3) They take more complex cost schemes regarding annotations and misclassifications into account. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.