# Active Hybrid Classification

arXiv: https://arxiv.org/abs/2101.08854 · HTML: https://mlsanity.com/paper/2101.08854/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 40 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 4 of 4)

- [Deep Active Learning for Named Entity Recognition](https://mlsanity.com/paper/1707.05928/) (2017)
- [Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages](https://mlsanity.com/paper/1605.05894/) (2016)
- [Crowd-based Multi-Predicate Screening of Papers in Literature Reviews](https://mlsanity.com/paper/1803.09814/) (2018)
- [Combining Crowd and Machines for Multi-predicate Item Screening](https://mlsanity.com/paper/1904.00714/) (2019)

## Abstract (arXiv)

Hybrid crowd-machine classifiers can achieve superior performance by combining the cost-effectiveness of automatic classification with the accuracy of human judgment. This paper shows how crowd and machines can support each other in tackling classification problems. Specifically, we propose an architecture that orchestrates active learning and crowd classification and combines them in a virtuous cycle. We show that when the pool of items to classify is finite we face learning vs. exploitation trade-off in hybrid classification, as we need to balance crowd tasks optimized for creating a training dataset with tasks optimized for classifying items in the pool. We define the problem, propose a set of heuristics and evaluate the approach on three real-world datasets with different characteristics in terms of machine and crowd classification performance, showing that our active hybrid approach significantly outperforms baselines.