mlsanityData to 7 Oct 2026

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

arXiv 2303.15056 ↗First posted 27 Mar 2023cs.CL cs.CYcited by 849

Not on the map. Why?

Cited by 39 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than 0.003 -- about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.

Key facts. Cited by 849 papers in the map (data through 7 Oct 2026). Builds on 8 of its 25 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

8 of 25 references.

References · 8

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  3. 2022Using cognitive psychology to understand GPT-3cited by 389
  4. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  5. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  6. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  7. 2023Large Language Models Can Be Used to Estimate the Latent Positions of Politicianscited by 33
  8. 2023Large Language Models as Corporate Lobbyistscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 182; 2024: 287; 2025: 183; 2026: 197.

849 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
38 more
  1. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  2. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  5. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  7. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  8. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  10. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  11. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  12. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  13. 2023Large language models can accurately predict searcher preferencescited by 129
  14. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  15. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  16. 2023Generative AIcited by 119
  17. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 115
  18. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  19. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  20. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  21. 2024Critique-out-Loud Reward Modelscited by 105
  22. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  23. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  24. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 97
  25. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  26. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  27. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  28. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  29. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  30. 2024The Political Preferences of LLMscited by 83
  31. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  32. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  33. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  34. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  35. 2023Automated Annotation with Generative AI Requires Validationcited by 75
  36. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  37. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  38. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73

50 most cited of 849 All 849 (JSON)