mlsanityData to 7 Oct 2026

The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation

arXiv 2405.01299 ↗First posted 2 May 2024cs.CL cs.AI cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have emerged as powerful support tools across various natural language tasks and a range of application domains. Recent studies focus on exploring their capabilities for data annotation. This paper provides a comparative overview of twelve studies investigating the potential of LLMs in labelling data. While the models demonstrate promising cost and time-saving benefits, there exist considerable limitations, such as representativeness, bias, sensitivity to prompt variations and English language preference. Leveraging insights from these studies, our empirical analysis further examines the alignment between human and GPT-generated opinion distributions across four subjective datasets. In contrast to the studies examining representation, our methodology directly obtains the opinion distribution from GPT. Our analysis thereby supports the minority of studies that are considering diverse perspectives when evaluating data annotation tasks and highlights the need for further research in this direction.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

28 of 37 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  9. 2023Whose Opinions Do Language Models Reflect?cited by 759
  10. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  11. 2022Is GPT-3 a Good Data Annotator?cited by 208
  12. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
16 more
  1. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  2. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  3. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  4. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  5. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  6. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  7. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  8. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  9. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  10. 2023LLMs Accelerate Annotation for Medical Information Extractioncited by 77
  11. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  12. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  13. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
  14. 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
  15. 2021Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detectioncited by 38
  16. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 6.

29 papers.

Sort:

By area:

  1. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  2. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  3. 2024Personas with Attitudes: Controlling LLMs for Diverse Data Annotationcited by 21
  4. 2024"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?cited by 16
  5. 2024MBIAS: Mitigating Bias in Large Language Models While Retaining Contextcited by 15
  6. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  7. 2024On Limitations of LLM as Annotator for Low Resource Languagescited by 12
  8. 2024A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLIcited by 8
  9. 2024Understanding The Effect Of Temperature On Alignment With Human Opinionscited by 4
  10. 2025Delving into Instance-Dependent Label Noise in Graph Data: A Comprehensive Study and Benchmarkcited by 4
  11. 2025Revisiting Active Learning under (Human) Label Variationcited by 4
  12. 2025Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanationcited by 4
17 more
  1. 2025Embracing Diversity: A Multi-Perspective Approach with Soft Labelscited by 3
  2. 2025WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thaicited by 3
  3. 2025The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrumcited by 3
  4. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Valuescited by 3
  5. 2026RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluationcited by 3
  6. 2025Aligning Black-box Language Models with Human Judgmentscited by 1
  7. 2025Analysis of LLM as a grammatical feature tagger for African American Englishcited by 1
  8. 2025Enhancing Multi-Label Emotion Analysis and Corresponding Intensities for Ethiopian Languagescited by 1
  9. 2025Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yetcited by 0
  10. 2025PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relationscited by 0
  11. 2025Exploring the Potential of Large Language Models in Fine-Grained Review Comment Classificationcited by 0
  12. 2025Generative Large Language Models (gLLMs) in Content Analysis: A Practical Guide for Communication Researchcited by 0
  13. 2026LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metricscited by 0
  14. 2026Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinesecited by 0
  15. 2026The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Predictioncited by 0
  16. 2026Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labelscited by 0
  17. 2026Extracting ontology-compliant knowledge from scientific text describing irradiated materials using large language modelscited by 0