# A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation

arXiv: https://arxiv.org/abs/2401.09760 · HTML: https://mlsanity.com/paper/2401.09760/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 8 of 8)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness](https://mlsanity.com/paper/2305.12947/) (2023)
- [Rank Aggregation via Heterogeneous Thurstone Preference Models](https://mlsanity.com/paper/1912.01211/) (2019)
- [Multiview Representation Learning from Crowdsourced Triplet Comparisons](https://mlsanity.com/paper/2302.03987/) (2023)

## Built on by (most cited first, 23 of 23)

- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information](https://mlsanity.com/paper/2510.01499/) (2025)
- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [Evaluation of an LLM in Identifying Logical Fallacies: A Call for Rigor When Adopting LLMs in HCI Research](https://mlsanity.com/paper/2404.05213/) (2024)
- [ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation](https://mlsanity.com/paper/2503.01052/) (2025)
- [Sanitizing Manufacturing Dataset Labels Using Vision-Language Models](https://mlsanity.com/paper/2506.23465/) (2025)
- [Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting](https://mlsanity.com/paper/2409.09615/) (2024)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations](https://mlsanity.com/paper/2410.17099/) (2024)
- [From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysis](https://mlsanity.com/paper/2501.06101/) (2025)
- [Bias in the Loop: How Humans Evaluate AI-Generated Suggestions](https://mlsanity.com/paper/2509.08514/) (2025)
- [Human-AI Complementarity: A Goal for Amplified Oversight](https://mlsanity.com/paper/2510.26518/) (2025)
- [Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models](https://mlsanity.com/paper/2407.10645/) (2024)
- [VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2409.18417/) (2024)
- [Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study](https://mlsanity.com/paper/2510.12835/) (2025)
- [Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation](https://mlsanity.com/paper/2511.01482/) (2025)
- [Towards Human-AI Accessibility Mapping in India: VLM-Guided Annotations and POI-Centric Analysis in Chandigarh](https://mlsanity.com/paper/2602.09216/) (2026)
- [Human-LLM Collaborative Construction of a Cantonese Emotion Lexicon](https://mlsanity.com/paper/2410.11526/) (2024)
- [Generating Privacy Stories From Software Documentation](https://mlsanity.com/paper/2506.23014/) (2025)
- [Understanding the Essence: Delving into Annotator Prototype Learning for Multi-Class Annotation Aggregation](https://mlsanity.com/paper/2508.02123/) (2025)
- [CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters](https://mlsanity.com/paper/2601.04885/) (2026)
- [AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling](https://mlsanity.com/paper/2607.18465/) (2026)

## Abstract (arXiv)

Whether Large Language Models (LLMs) can outperform crowdsourcing on the data annotation task is attracting interest recently. Some works verified this issue with the average performance of individual crowd workers and LLM workers on some specific NLP tasks by collecting new datasets. However, on the one hand, existing datasets for the studies of annotation quality in crowdsourcing are not yet utilized in such evaluations, which potentially provide reliable evaluations from a different viewpoint. On the other hand, the quality of these aggregated labels is crucial because, when utilizing crowdsourcing, the estimated labels aggregated from multiple crowd labels to the same instances are the eventually collected labels. Therefore, in this paper, we first investigate which existing crowdsourcing datasets can be used for a comparative study and create a benchmark. We then compare the quality between individual crowd labels and LLM labels and make the evaluations on the aggregated labels. In addition, we propose a Crowd-LLM hybrid label aggregation method and verify the performance. We find that adding LLM labels from good LLMs to existing crowdsourcing datasets can enhance the quality of the aggregated labels of the datasets, which is also higher than the quality of LLM labels themselves.