# Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks

arXiv: https://arxiv.org/abs/2507.15821 · HTML: https://mlsanity.com/paper/2507.15821/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2025.

## Built on by (most cited first, 21 of 21)

- [Stop Automating Peer Review Without Rigorous Evaluation](https://mlsanity.com/paper/2605.03202/) (2026)
- [Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub](https://mlsanity.com/paper/2601.00477/) (2026)
- ["It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models](https://mlsanity.com/paper/2511.08917/) (2025)
- [Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks](https://mlsanity.com/paper/2603.05801/) (2026)
- [Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer](https://mlsanity.com/paper/2606.11018/) (2026)
- [EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footage](https://mlsanity.com/paper/2607.06468/) (2026)
- [Structured Exploration and Exploitation of Label Functions for Automated Data Annotation](https://mlsanity.com/paper/2604.08578/) (2026)
- [What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems](https://mlsanity.com/paper/2605.20548/) (2026)
- [SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments](https://mlsanity.com/paper/2607.05264/) (2026)
- [Interactive In-Meeting Speaker Correction with Human Feedback](https://mlsanity.com/paper/2509.18377/) (2025)
- [Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique](https://mlsanity.com/paper/2601.09905/) (2026)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://mlsanity.com/paper/2601.13729/) (2026)
- [AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions](https://mlsanity.com/paper/2603.01773/) (2026)
- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)
- [Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation](https://mlsanity.com/paper/2605.25781/) (2026)
- [WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification](https://mlsanity.com/paper/2605.26070/) (2026)
- [Telenor Nordics Customer Service self-help corpus](https://mlsanity.com/paper/2605.26891/) (2026)
- [GlossAssist -- A Tool to Simplify Corpus Creation and Study the Effect of NLP Models in Low-Resource Documentation Settings](https://mlsanity.com/paper/2606.04367/) (2026)
- [From Detection to Characterization: A Large-Scale Study of Ragebait on Japanese X](https://mlsanity.com/paper/2609.02262/) (2026)
- [CALICO: A Human-Centered, Codebook-Aligned System for Annotation](https://mlsanity.com/paper/2609.14726/) (2026)
- [Canonical Procedural Actions: An Auditable Annotation Protocol for Tool-Use Agent Traces](https://mlsanity.com/paper/2609.24264/) (2026)

## Abstract (arXiv)

LLM use in annotation is becoming widespread, and given LLMs' overall promising performance and speed, simply "reviewing" LLM annotations in interpretive tasks can be tempting. In subjective annotation tasks with multiple plausible answers, reviewing LLM outputs can change the label distribution, impacting both the evaluation of LLM performance, and analysis using these labels in a social science task downstream. We conducted a pre-registered experiment with 410 unique annotators and over 7,000 annotations testing three AI assistance conditions against controls, using two models, and two datasets. We find that presenting crowdworkers with LLM-generated annotation suggestions did not make them faster, but did improve their self-reported confidence in the task. More importantly, annotators strongly took the LLM suggestions, significantly changing the label distribution compared to the baseline. When these labels created with LLM assistance are used to evaluate LLM performance, reported model performance significantly increases. We believe our work underlines the importance of understanding the impact of LLM-assisted annotation on subjective, qualitative tasks, on the creation of gold data for training and testing, and on the evaluation of NLP systems on subjective tasks.