# If in a Crowdsourced Data Annotation Pipeline, a GPT-4

arXiv: https://arxiv.org/abs/2402.16795 · HTML: https://mlsanity.com/paper/2402.16795/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 72 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation](https://mlsanity.com/paper/2206.08931/) (2022)
- [ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness](https://mlsanity.com/paper/2305.12947/) (2023)
- [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023)
- [Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning](https://mlsanity.com/paper/2307.02179/) (2023)
- [CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweets](https://mlsanity.com/paper/2204.12164/) (2022)
- [LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs](https://mlsanity.com/paper/2307.10168/) (2023)
- [Learning from Crowds with Crowd-Kit](https://mlsanity.com/paper/2109.08584/) (2021)
- [A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation](https://mlsanity.com/paper/2401.09760/) (2024)
- [Adversarial Crowdsourcing Through Robust Rank-One Matrix Completion](https://mlsanity.com/paper/2010.12181/) (2020)
- [Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions](https://mlsanity.com/paper/1801.02546/) (2018)
- [BioTABQA: Instruction Learning for Biomedical Table Question Answering](https://mlsanity.com/paper/2207.02419/) (2022)
- [Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing](https://mlsanity.com/paper/2105.12762/) (2021)
- [MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluation](https://mlsanity.com/paper/2308.08147/) (2023)
- [Empirical Methodology for Crowdsourcing Ground Truth](https://mlsanity.com/paper/1809.08888/) (2018)
- [On the state of reporting in crowdsourcing experiments and a checklist to aid current practices](https://mlsanity.com/paper/2107.13519/) (2021)
- [Good Data, Large Data, or No Data? Comparing Three Approaches in Developing Research Aspect Classifiers for Biomedical Papers](https://mlsanity.com/paper/2306.04820/) (2023)

## Built on by (most cited first, 30 of 38)

- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Neither Valid nor Reliable? Investigating the Use of LLMs as Judges](https://mlsanity.com/paper/2508.18076/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://mlsanity.com/paper/2407.04693/) (2024)
- [Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions](https://mlsanity.com/paper/2405.16402/) (2024)
- [Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs](https://mlsanity.com/paper/2408.10902/) (2024)
- [ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models](https://mlsanity.com/paper/2403.20262/) (2024)
- [ARTICLE: Annotator Reliability Through In-Context Learning](https://mlsanity.com/paper/2409.12218/) (2024)
- [Using LLMs to Establish Implicit User Sentiment of Software Desirability](https://mlsanity.com/paper/2408.01527/) (2024)
- [Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations](https://mlsanity.com/paper/2410.17099/) (2024)
- [Large Language Models for Automatic Detection of Sensitive Topics](https://mlsanity.com/paper/2409.00940/) (2024)
- [Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs](https://mlsanity.com/paper/2409.10702/) (2024)
- [ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering](https://mlsanity.com/paper/2507.00828/) (2025)
- [Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification](https://mlsanity.com/paper/2507.05010/) (2025)
- [Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification](https://mlsanity.com/paper/2403.05640/) (2024)
- [Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysis](https://mlsanity.com/paper/2405.05758/) (2024)
- [How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse](https://mlsanity.com/paper/2512.19903/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [What Color Scheme is More Effective in Assisting Readers to Locate Information in a Color-Coded Article?](https://mlsanity.com/paper/2408.06494/) (2024)
- [Speech Foundation Models and Crowdsourcing for Efficient, High-Quality Data Collection](https://mlsanity.com/paper/2412.11978/) (2024)
- [Augmenting Image Annotation: A Human-LMM Collaborative Framework for Efficient Object Selection and Label Generation](https://mlsanity.com/paper/2503.11096/) (2025)
- [From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models](https://mlsanity.com/paper/2503.20715/) (2025)
- [OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering](https://mlsanity.com/paper/2512.15979/) (2025)
- [Probabilistic Label Spreading: Efficient and Consistent Estimation of Soft Labels with Epistemic Uncertainty on Graphs](https://mlsanity.com/paper/2602.04574/) (2026)
- [When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speech](https://mlsanity.com/paper/2603.25269/) (2026)
- [Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates](https://mlsanity.com/paper/2608.29455/) (2026)
- [Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet](https://mlsanity.com/paper/2502.05291/) (2025)
- [Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models](https://mlsanity.com/paper/2503.21000/) (2025)

## Abstract (arXiv)

Recent studies indicated GPT-4 outperforms online crowd workers in data labeling accuracy, notably workers from Amazon Mechanical Turk (MTurk). However, these studies were criticized for deviating from standard crowdsourcing practices and emphasizing individual workers' performances over the whole data-annotation process. This paper compared GPT-4 and an ethical and well-executed MTurk pipeline, with 415 workers labeling 3,177 sentence segments from 200 scholarly articles using the CODA-19 scheme. Two worker interfaces yielded 127,080 labels, which were then used to infer the final labels through eight label-aggregation algorithms. Our evaluation showed that despite best practices, MTurk pipeline's highest accuracy was 81.5%, whereas GPT-4 achieved 83.6%. Interestingly, when combining GPT-4's labels with crowd labels collected via an advanced worker interface for aggregation, 2 out of the 8 algorithms achieved an even higher accuracy (87.5%, 87.0%). Further analysis suggested that, when the crowd's and GPT-4's labeling strengths are complementary, aggregating them could increase labeling accuracy.