mlsanityData to 14 Oct 2026

ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

arXiv 2304.06588 ↗First posted 13 Apr 2023cs.CL cs.AI cs.SIcited by 100

Not on the map. Why?

Abstract (arXiv)

This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet. The LLM is compared to manual annotation by both expert classifiers and crowd workers, generally considered the gold standard for such tasks. We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy. The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers. The LLM is able to correctly annotate messages that require reasoning on the basis of contextual knowledge, and inferences around the author's intentions - traditionally seen as uniquely human abilities. These findings suggest that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 8 of its 25 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

8 of 25 references.

References · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 34; 2024: 36; 2025: 16; 2026: 14.

100 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  3. 2023Large language models can accurately predict searcher preferencescited by 131
  4. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  5. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  6. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  7. 2023Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithmscited by 65
  8. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  9. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  10. 2023ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relationscited by 40
  11. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 39
  12. 2023Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?cited by 39
38 more
  1. 2024Best Practices for Text Annotation with Large Language Modelscited by 36
  2. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
  3. 2023LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMscited by 27
  4. 2024EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretationcited by 27
  5. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 27
  6. 2024A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregationcited by 23
  7. 2023Using Large Language Models for Qualitative Analysis can Introduce Serious Biascited by 22
  8. 2023Stance Detection: A Practical Guide to Classifying Political Beliefs in Textcited by 20
  9. 2024Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysiscited by 20
  10. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  11. 2023Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Sciencecited by 19
  12. 2023Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformationcited by 19
  13. 2024Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiencycited by 18
  14. 2023Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreementcited by 17
  15. 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 17
  16. 2023Does ChatGPT have Theory of Mind?cited by 16
  17. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
  18. 2023LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPTcited by 15
  19. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 15
  20. 2024LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generationcited by 15
  21. 2024Prompt Stability Scoring for Text Annotation with Large Language Modelscited by 15
  22. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  23. 2023Watch Your Language: Investigating Content Moderation with Large Language Modelscited by 14
  24. 2023A Large Language Model Approach to Educational Survey Feedback Analysiscited by 14
  25. 2023Positioning Political Texts with Large Language Models by Asking and Averagingcited by 13
  26. 2024Web Retrieval Agents for Evidence-Based Misinformation Detectioncited by 13
  27. 2023Revisiting Prompt Engineering via Declarative Crowdsourcingcited by 12
  28. 2023Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Modelscited by 12
  29. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Datacited by 12
  30. 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
  31. 2023DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queriescited by 11
  32. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 11
  33. 2023Prompting a Large Language Model to Generate Diverse Motivational Messages: A Comparison with Human-Written Messagescited by 10
  34. 2024LLMs Among Us: Generative AI Participating in Digital Discoursecited by 9
  35. 2024LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domainscited by 9
  36. 2024Harmful YouTube Video Detection: A Taxonomy of Online Harm and MLLMs as Alternative Annotatorscited by 8
  37. 2024Emojis Decoded: Leveraging ChatGPT for Enhanced Understanding in Social Media Communicationscited by 7
  38. 2024Automated Review Generation Method Based on Large Language Modelscited by 7

50 most cited of 100 All 100 (JSON)