mlsanityData to 14 Oct 2026

Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks

arXiv 2304.10145 ↗First posted 20 Apr 2023cs.AI cs.CLcited by 89

Not on the map. Why?

Abstract (arXiv)

The release of ChatGPT has uncovered a range of possibilities whereby large language models (LLMs) can substitute human intelligence. In this paper, we seek to understand whether ChatGPT has the potential to reproduce human-generated label annotations in social computing tasks. Such an achievement could significantly reduce the cost and complexity of social computing research. As such, we use ChatGPT to relabel five seminal datasets covering stance detection (2x), sentiment analysis, hate speech, and bot detection. Our results highlight that ChatGPT does have the potential to handle these data annotation tasks, although a number of challenges remain. ChatGPT obtains an average accuracy 0.609. Performance is highest for the sentiment analysis dataset, with ChatGPT correctly annotating 64.9% of tweets. Yet, we show that performance varies substantially across individual labels. We believe this work can open up new lines of analysis and act as a basis for future research into the exploitation of ChatGPT for human annotation tasks.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 21 of its 47 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

21 of 47 references.

References · 21

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  3. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  5. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 217
  6. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  7. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  8. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  9. 2020TweepFake: about Detecting Deepfake Tweetscited by 117
  10. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 110
  11. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  12. 2022How would Stance Detection Techniques Evolve after the Launch of ChatGPT?cited by 76
9 more
  1. 2018Thou shalt not hate: Countering Online Hate Speechcited by 64
  2. 2023Can we trust the evaluation on ChatGPT?cited by 63
  3. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  4. 2020Racism is a Virus: Anti-Asian Hate and Counterspeech in Social Media during the COVID-19 Crisiscited by 40
  5. 2020A Survey on Large-scale Machine Learningcited by 35
  6. 2017Unsupervised Machine Learning for Networking: Techniques, Applications and Research Challengescited by 12
  7. 2023Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Datacited by 7
  8. 2022Twitter Dataset for 2022 Russo-Ukrainian Crisiscited by 6
  9. 2022A Reddit Dataset for the Russo-Ukrainian Conflict in 2022cited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 40; 2025: 16; 2026: 3.

89 papers.

Sort:

By area:

  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  2. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  3. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  5. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  6. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  7. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  8. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 70
  9. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  10. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  11. 2023Probing LLMs for hate speech detection: strengths and vulnerabilitiescited by 52
  12. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
38 more
  1. 2023SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Supportcited by 46
  2. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  3. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  4. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  5. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  6. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  7. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
  8. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  9. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  10. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  11. 2024A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregationcited by 23
  12. 2024CodeMirage: Hallucinations in Code Generated by Large Language Modelscited by 23
  13. 2024MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Modelscited by 21
  14. 2023Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformationcited by 19
  15. 2023SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understandingcited by 18
  16. 2024ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?cited by 17
  17. 2024On the Brittle Foundations of ReAct Prompting for Agentic Large Language Modelscited by 17
  18. 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 17
  19. 2024Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationalescited by 16
  20. 2023Leveraging Large Language Models for Automated Dialogue Analysiscited by 14
  21. 2024A Survey on Human-Centric LLMscited by 13
  22. 2023AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topicscited by 12
  23. 2023Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Modelscited by 12
  24. 2024Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluationcited by 12
  25. 2024On Limitations of LLM as Annotator for Low Resource Languagescited by 12
  26. 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
  27. 2024LLM Chain Ensembles for Scalable and Accurate Data Annotationcited by 11
  28. 2024DepressionEmo: A novel dataset for multilabel classification of depression emotionscited by 10
  29. 2024Representation Bias in Political Sample Simulations with Large Language Modelscited by 10
  30. 2025Evaluating Large Language Models as Expert Annotatorscited by 9
  31. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
  32. 2023Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweetscited by 8
  33. 2023HateRephrase: Zero- and Few-Shot Reduction of Hate Intensity in Online Posts using Large Language Modelscited by 8
  34. 2024Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learningcited by 8
  35. 2023AGI: Artificial General Intelligence for Educationcited by 7
  36. 2024Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensivenesscited by 7
  37. 2023Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detectioncited by 6
  38. 2024Investigating Annotator Bias in Large Language Models for Hate Speech Detectioncited by 6

50 most cited of 89 All 89 (JSON)