mlsanityData to 7 Oct 2026

If in a Crowdsourced Data Annotation Pipeline, a GPT-4

arXiv 2402.16795 ↗First posted 26 Feb 2024cs.HC cs.AI cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent studies indicated GPT-4 outperforms online crowd workers in data labeling accuracy, notably workers from Amazon Mechanical Turk (MTurk). However, these studies were criticized for deviating from standard crowdsourcing practices and emphasizing individual workers' performances over the whole data-annotation process. This paper compared GPT-4 and an ethical and well-executed MTurk pipeline, with 415 workers labeling 3,177 sentence segments from 200 scholarly articles using the CODA-19 scheme. Two worker interfaces yielded 127,080 labels, which were then used to infer the final labels through eight label-aggregation algorithms. Our evaluation showed that despite best practices, MTurk pipeline's highest accuracy was 81.5%, whereas GPT-4 achieved 83.6%. Interestingly, when combining GPT-4's labels with crowd labels collected via an advanced worker interface for aggregation, 2 out of the 8 algorithms achieved an even higher accuracy (87.5%, 87.0%). Further analysis suggested that, when the crowd's and GPT-4's labeling strengths are complementary, aggregating them could increase labeling accuracy.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 25 of its 72 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

25 of 72 references.

References · 25

  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  2. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  3. 2022Data Contamination: From Memorization to Exploitationcited by 234
  4. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  5. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  6. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  7. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  8. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  9. 2014Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcingcited by 63
  10. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  11. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  12. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
13 more
  1. 2023Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuningcited by 31
  2. 2022CoVERT: A Corpus of Fact-checked Biomedical COVID-19 Tweetscited by 29
  3. 2023LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMscited by 27
  4. 2021Learning from Crowds with Crowd-Kitcited by 23
  5. 2024A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregationcited by 23
  6. 2020Adversarial Crowdsourcing Through Robust Rank-One Matrix Completioncited by 21
  7. 2018Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actionscited by 16
  8. 2022BioTABQA: Instruction Learning for Biomedical Table Question Answeringcited by 13
  9. 2021Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcingcited by 11
  10. 2023MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluationcited by 10
  11. 2018Empirical Methodology for Crowdsourcing Ground Truthcited by 7
  12. 2021On the state of reporting in crowdsourcing experiments and a checklist to aid current practicescited by 6
  13. 2023Good Data, Large Data, or No Data? Comparing Three Approaches in Developing Research Aspect Classifiers for Biomedical Paperscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 11.

38 papers.

Sort:

By area:

  1. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  2. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  3. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  4. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  5. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  6. 2024ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Modelscited by 14
  7. 2024Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactionscited by 13
  8. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  9. 2024ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Modelscited by 9
  10. 2024ARTICLE: Annotator Reliability Through In-Context Learningcited by 6
  11. 2024Using LLMs to Establish Implicit User Sentiment of Software Desirabilitycited by 4
  12. 2024Human-LLM Hybrid Text Answer Aggregation for Crowd Annotationscited by 4
26 more
  1. 2024Large Language Models for Automatic Detection of Sensitive Topicscited by 3
  2. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3
  3. 2025ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clusteringcited by 3
  4. 2025Co-DETECT: Collaborative Discovery of Edge Cases in Text Classificationcited by 3
  5. 2024Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classificationcited by 2
  6. 2024Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysiscited by 2
  7. 2025How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discoursecited by 2
  8. 2024What Color Scheme is More Effective in Assisting Readers to Locate Information in a Color-Coded Article?cited by 1
  9. 2024Speech Foundation Models and Crowdsourcing for Efficient, High-Quality Data Collectioncited by 1
  10. 2025Augmenting Image Annotation: A Human-LMM Collaborative Framework for Efficient Object Selection and Label Generationcited by 1
  11. 2025From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Modelscited by 1
  12. 2025OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineeringcited by 1
  13. 2026Probabilistic Label Spreading: Efficient and Consistent Estimation of Soft Labels with Epistemic Uncertainty on Graphscited by 1
  14. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  15. 2026When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speechcited by 1
  16. 2025Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yetcited by 0
  17. 2025Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Modelscited by 0
  18. 2025A Longitudinal Study on Different Annotator Feedback Loops in Complex RAG Taskscited by 0
  19. 2026The Critical Role of Aspects in Measuring Document Similaritycited by 0
  20. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  21. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  22. 2026Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Modelcited by 0
  23. 2026Evaluating LLM Usage for Efficient and Explainable Numerical and Classified Implicit Sentiment Analysis of Product Desirabilitycited by 0
  24. 2026Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogatescited by 0
  25. 2026Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotationcited by 0
  26. 2026Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?cited by 0