mlsanityData to 14 Oct 2026

The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks

arXiv 2304.13861 ↗First posted 26 Apr 2023cs.CL cs.CY physics.soc-phcited by 35

Not on the map. Why?

Abstract (arXiv)

In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks of varying complexity. Additionally, we examine the impact of training data sizes on performance. Our findings reveal that models trained on human-labeled data consistently exhibit superior or comparable performance compared to their synthetically augmented counterparts. Nevertheless, synthetic augmentation proves beneficial, particularly in improving performance on rare classes within multi-class tasks. Furthermore, we leverage GPT-4 and Llama-2 for zero-shot classification and find that, while they generally display strong performance, they often fall short when compared to specialized classifiers trained on moderately sized training sets.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 27 of its 50 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

27 of 50 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  9. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  11. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  12. 2022Is GPT-3 a Good Data Annotator?cited by 208
15 more
  1. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  2. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 187
  3. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  4. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  5. 2022How would Stance Detection Techniques Evolve after the Launch of ChatGPT?cited by 76
  6. 2018Modeling Empathy and Distress in Reaction to News Storiescited by 52
  7. 2019Offensive Language and Hate Speech Detection for Danishcited by 49
  8. 2020GenAug: Data Augmentation for Finetuning Text Generatorscited by 41
  9. 2020Quantifying Intimacy in Languagecited by 25
  10. 2021Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifierscited by 24
  11. 2021Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexicacited by 23
  12. 2019TalkDown: A Corpus for Condescension Detection in Contextcited by 21
  13. 2020Ten Social Dimensions of Conversations and Relationshipscited by 18
  14. 2021MOVER: Mask, Over-generate and Rank for Hyperbole Generationcited by 13
  15. 2022Improving Short Text Classification With Augmented Data Using GPT-3cited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 11; 2025: 14; 2026: 2.

35 papers.

Sort:

By area:

  1. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 65
  2. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Sciencecited by 37
  3. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  4. 2023People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detectioncited by 25
  5. 2023NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violationcited by 24
  6. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 24
  7. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  8. 2024I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responsescited by 15
  9. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectivecited by 15
  10. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  11. 2026Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondentscited by 10
  12. 2025What Has Been Lost with Synthetic Evaluation?cited by 9
23 more
  1. 2023Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweetscited by 8
  2. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systemscited by 8
  3. 2024AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responsescited by 6
  4. 2023BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptionscited by 5
  5. 2024Graph Neural Backdoor: Fundamentals, Methodologies, Applications, and Future Directionscited by 4
  6. 2025Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applicationscited by 4
  7. 2025Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detectioncited by 4
  8. 2024PGA-SciRE: Harnessing LLM on Data Augmentation for Enhancing Scientific Relation Extractioncited by 3
  9. 2024Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Textscited by 3
  10. 2025Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialoguecited by 3
  11. 2025LACA: Improving Cross-lingual Aspect-Based Sentiment Analysis with LLM Data Augmentationcited by 3
  12. 2023Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generationcited by 2
  13. 2025SynQuE: Estimating Synthetic Dataset Quality Without Annotationscited by 2
  14. 2025Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inferencecited by 1
  15. 2025Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognitioncited by 1
  16. 2025Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challengescited by 1
  17. 2025GRAID: Synthetic Data Generation with Geometric Constraints and Multi-Agentic Reflection for Harmful Content Detectioncited by 1
  18. 2024From Measurement Instruments to Data: Leveraging Theory-Driven Synthetic Training Data for Classifying Social Constructscited by 0
  19. 2025ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Linkscited by 0
  20. 2025MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Datacited by 0
  21. 2025When Distributions Shifts: Causal Generalization for Low-Resource Languagescited by 0
  22. 2025Time Travel: LLM-Assisted Semantic Behavior Localization with Git Bisectcited by 0
  23. 2026When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classificationcited by 0