# Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interface

arXiv: https://arxiv.org/abs/2104.04122 · HTML: https://mlsanity.com/paper/2104.04122/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2021.

## Built on by (most cited first, 21 of 21)

- [Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction](https://mlsanity.com/paper/2303.09224/) (2023)
- [Human-AI collaboration is not very collaborative yet: A taxonomy of interaction patterns in AI-assisted decision making from a systematic review](https://mlsanity.com/paper/2310.19778/) (2023)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Making](https://mlsanity.com/paper/2401.07058/) (2024)
- [Decoding AI's Nudge: A Unified Framework to Predict Human Behavior in AI-assisted Decision Making](https://mlsanity.com/paper/2401.05840/) (2024)
- [A Test for Evaluating Performance in Human-Computer Systems](https://mlsanity.com/paper/2206.12390/) (2022)
- [Time-Efficient Reward Learning via Visually Assisted Cluster Ranking](https://mlsanity.com/paper/2212.00169/) (2022)
- [AI Simulation by Digital Twins: Systematic Survey, Reference Framework, and Mapping to a Standardized Architecture](https://mlsanity.com/paper/2506.06580/) (2025)
- [Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking](https://mlsanity.com/paper/2404.18881/) (2024)
- [What Makes An Expert? Reviewing How ML Researchers Define "Expert"](https://mlsanity.com/paper/2411.00179/) (2024)
- [Learning to Verify Summary Facts with Fine-Grained LLM Feedback](https://mlsanity.com/paper/2412.10689/) (2024)
- [DeepShovel: An Online Collaborative Platform for Data Extraction in Geoscience Literature with AI Assistance](https://mlsanity.com/paper/2202.10163/) (2022)
- [Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality](https://mlsanity.com/paper/2305.12811/) (2023)
- [Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs](https://mlsanity.com/paper/2409.10702/) (2024)
- [VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labels](https://mlsanity.com/paper/2507.09008/) (2025)
- [Optimising Human-Machine Collaboration for Efficient High-Precision Information Extraction from Text Documents](https://mlsanity.com/paper/2302.09324/) (2023)
- [Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks](https://mlsanity.com/paper/2507.02819/) (2025)
- [ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization](https://mlsanity.com/paper/2604.10520/) (2026)
- [Developing A Visual-Interactive Interface for Electronic Health Record Labeling: An Explainable Machine Learning Approach](https://mlsanity.com/paper/2209.12778/) (2022)
- [AI-Guided Defect Detection Techniques to Model Single Crystal Diamond Growth](https://mlsanity.com/paper/2404.07306/) (2024)
- [AI-Guided Feature Segmentation Techniques to Model Features from Single Crystal Diamond Growth](https://mlsanity.com/paper/2404.08017/) (2024)

## Abstract (arXiv)

Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has shown that providing AI assistance can improve the accuracy of binary decision tasks. However, the role of AI assistance in more complex data-labeling scenarios with a larger set of labels has not yet been explored. We designed an AI labeling assistant that uses a semi-supervised learning algorithm to predict the most probable labels for each example. We leverage these predictions to provide assistance in two ways: (i) providing a label recommendation and (ii) reducing the labeler's decision space by focusing their attention on only the most probable labels. We conducted a user study (n=54) to evaluate an AI-assisted interface for data labeling in this context. Our results highlight that the AI assistance improves both labeler accuracy and speed, especially when the labeler finds the correct label in the reduced label space. We discuss findings related to the presentation of AI assistance and design implications for intelligent labeling interfaces.