# Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning

arXiv: https://arxiv.org/abs/2007.03177 · HTML: https://mlsanity.com/paper/2007.03177/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 21 of 21)

- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [ImageInWords: Unlocking Hyper-Detailed Image Descriptions](https://mlsanity.com/paper/2405.02793/) (2024)
- [The Oscars of AI Theater: A Survey on Role-Playing with Language Models](https://mlsanity.com/paper/2407.11484/) (2024)
- [Compositional preference models for aligning LMs](https://mlsanity.com/paper/2310.13011/) (2023)
- [Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback](https://mlsanity.com/paper/2401.11458/) (2024)
- [Dissecting Human and LLM Preferences](https://mlsanity.com/paper/2402.11296/) (2024)
- [RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation](https://mlsanity.com/paper/2501.08617/) (2025)
- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents](https://mlsanity.com/paper/2405.12900/) (2024)
- [Towards Reliable Dermatology Evaluation Benchmarks](https://mlsanity.com/paper/2309.06961/) (2023)
- [Conditions on Preference Relations that Guarantee the Existence of Optimal Policies](https://mlsanity.com/paper/2311.01990/) (2023)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [TransClean: Finding False Positives in Multi-Source Entity Matching under Real-World Conditions via Transitive Consistency](https://mlsanity.com/paper/2506.04006/) (2025)
- [XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering](https://mlsanity.com/paper/2508.16139/) (2025)
- [Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program](https://mlsanity.com/paper/2310.05286/) (2023)
- [ORIS: Online Active Learning Using Reinforcement Learning-based Inclusive Sampling for Robust Streaming Analytics System](https://mlsanity.com/paper/2411.18060/) (2024)
- [From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse](https://mlsanity.com/paper/2509.14712/) (2025)
- [Agile Deliberation: Concept Deliberation for Subjective Visual Classification](https://mlsanity.com/paper/2512.10821/) (2025)
- [Architecture-agnostic Lipschitz-constant Bayesian header and its application to resolve semantically proximal classification errors with vision transformers](https://mlsanity.com/paper/2605.05908/) (2026)
- [An offline approach to fNIRS-guided reinforcement learning for robot behavior](https://mlsanity.com/paper/2607.14393/) (2026)

## Abstract (arXiv)

High-quality human annotations are necessary for creating effective machine learning-driven stream processing systems. We study hybrid stream processing systems based on a Human-In-The-Loop Machine Learning (HITL-ML) paradigm, in which one or many human annotators and an automatic classifier (trained at least partially by the human annotators) label an incoming stream of instances. This is typical of many near-real-time social media analytics and web applications, including annotating social media posts during emergencies by digital volunteer groups. From a practical perspective, low-quality human annotations result in wrong labels for retraining automated classifiers and indirectly contribute to the creation of inaccurate classifiers. Considering human annotation as a psychological process allows us to address these limitations. We show that human annotation quality is dependent on the ordering of instances shown to annotators and can be improved by local changes in the instance sequence/order provided to the annotators, yielding a more accurate annotation of the stream. We adapt a theoretically-motivated human error framework of mistakes and slips for the human annotation task to study the effect of ordering instances (i.e., an "annotation schedule"). Further, we propose an error-avoidance approach to the active learning paradigm for stream processing applications robust to these likely human errors (in the form of slips) when deciding a human annotation schedule. We support the human error framework using crowdsourcing experiments and evaluate the proposed algorithm against standard baselines for active learning via extensive experimentation on classification tasks of filtering relevant social media posts during natural disasters.