mlsanityData to 14 Oct 2026

Automated Annotation with Generative AI Requires Validation

arXiv 2306.00176 ↗First posted 31 May 2023cs.CL cs.AIcited by 76

Not on the map. Why?

Abstract (arXiv)

Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures, but their performance varies across annotation tasks due to prompt quality, text data idiosyncrasies, and conceptual difficulty. Because these challenges will persist even as LLM technology improves, we argue that any automated annotation process using an LLM must validate the LLM's performance against labels generated by humans. To this end, we outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. Using GPT-4, we validate this approach by replicating 27 annotation tasks across 11 datasets from recent social science articles in high-impact journals. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis. We make available easy-to-use software designed to implement our workflow and streamline the deployment of LLMs for automated annotation.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 9 of its 22 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

9 of 22 references.

References · 9

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  3. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  4. 2022Is GPT-3 a Good Data Annotator?cited by 208
  5. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  6. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  7. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  8. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  9. 2023On the rise of fear speech in online social mediacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 22; 2025: 18; 2026: 28.

76 papers.

Sort:

By area:

  1. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  3. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  4. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  5. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 42
  6. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 42
  7. 2024Best Practices for Text Annotation with Large Language Modelscited by 36
  8. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  9. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  10. 2025A Comprehensive Survey of Synthetic Tabular Data Generationcited by 28
  11. 2023Embedding Democratic Values into Social Media AIs via Societal Objective Functionscited by 25
  12. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
38 more
  1. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 23
  2. 2024STICKERCONV: Generating Multimodal Empathetic Responses from Scratchcited by 22
  3. 2024Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancycited by 20
  4. 2023A Large Language Model Approach to Educational Survey Feedback Analysiscited by 14
  5. 2023ContraDoc: Understanding Self-Contradictions in Documents with Large Language Modelscited by 12
  6. 2023Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Modelscited by 12
  7. 2026Evaluating Decision Models for Text Annotation in Computational Social Sciencecited by 11
  8. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 10
  9. 2025A Simple Ensemble Strategy for LLM Inference: Towards More Stable Text Classificationcited by 8
  10. 2024LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languagescited by 7
  11. 2024OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistantscited by 7
  12. 2024Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomenacited by 5
  13. 2024Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting: A Study on Factors Affecting Restaurant Evaluationcited by 5
  14. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluationcited by 5
  15. 2024Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Promptingcited by 5
  16. 2023LLM-enhanced Self-training for Cross-domain Constituency Parsingcited by 4
  17. 2024GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Textcited by 4
  18. 2024Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Languagecited by 4
  19. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understandingcited by 4
  20. 2026LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gapscited by 4
  21. 2024LLMs for Targeted Sentiment in News Headlines: Exploring the Descriptive-Prescriptive Dilemmacited by 3
  22. 2025Co-DETECT: Collaborative Discovery of Edge Cases in Text Classificationcited by 3
  23. 2025Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciencescited by 2
  24. 2026What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Mediacited by 2
  25. 2026Characterizing Narrative Content in Web-scale LLM Pretraining Datacited by 2
  26. 2026Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructscited by 2
  27. 2024Utilizing Large Language Models to Identify Reddit Users Considering Vaping Cessation for Digital Interventionscited by 1
  28. 2025Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inferencecited by 1
  29. 2025Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciencescited by 1
  30. 2025Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Datacited by 1
  31. 2025Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applicationscited by 1
  32. 2025Structuring Radiology Reports: Challenging LLMs with Lightweight Modelscited by 1
  33. 2025How Persuasive is Your Context?cited by 1
  34. 2026LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Textcited by 1
  35. 2026Adversarial Arena: Crowdsourcing Data Generation through Interactive Competitioncited by 1
  36. 2026How Much Does Persuasion Strategy Matter? LLM-Annotated Evidence from Charitable Donation Dialoguescited by 1
  37. 2026Conduct Under Pressure: What Sixty Language Models Do When a User Pushescited by 1
  38. 2024Cross-domain Chinese Sentence Pattern Parsingcited by 0

50 most cited of 76 All 76 (JSON)