mlsanityData to 7 Oct 2026

Are Expert-Level Language Models Expert-Level Annotators?

arXiv 2410.03254 ↗First posted 4 Oct 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of works have reported positive results on leveraging LLMs as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we investigate comprehensive approaches across three highly specialized domains and discuss practical suggestions from a cost-effectiveness perspective. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

27 of 37 references.

References · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  12. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
15 more
  1. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  2. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  3. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  4. 2022Is GPT-3 a Good Data Annotator?cited by 208
  5. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  6. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  7. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  8. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  9. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  10. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  11. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 73
  12. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  13. 2023Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Examscited by 18
  14. 2022Hierarchical Annotation for Building A Suite of Clinical Natural Language Processing Tasks: Progress Note Understandingcited by 13
  15. 2024GPTs Are Multilingual Annotators for Sequence Generation Taskscited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinkingcited by 3
  3. 2025How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAGcited by 2