mlsanityData to 14 Oct 2026

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

arXiv 2303.16854 ↗First posted 29 Mar 2023cs.CLcited by 172

Not on the map. Why?

Abstract (arXiv)

Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance. However, data annotation is time-consuming and expensive, especially when the task involves a large amount of data or requires specialized domains. Recently, GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator when provided with sufficient guidance and demonstrated examples. Accordingly, we propose AnnoLLM, an annotation system powered by LLMs, which adopts a two-step approach, explain-then-annotate. Concretely, we first prompt LLMs to provide explanations for why the specific ground truth answer/label was assigned for a given example. Then, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data with LLMs. Our experiment results on three tasks, including user input and keyword relevance assessment, BoolQ, and WiC, demonstrate that AnnoLLM surpasses or performs on par with crowdsourced annotators. Furthermore, we build the first conversation-based information retrieval dataset employing AnnoLLM. This dataset is designed to facilitate the development of retrieval models capable of retrieving pertinent documents for conversational text. Human evaluation has validated the dataset's high quality.

Key facts. Cited by 172 papers in the map (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

28 of 50 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
16 more
  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  7. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  8. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  11. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  12. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  13. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  14. 2018Style Transfer Through Back-Translationcited by 124
  15. 2020Evidence-based Factual Error Correctioncited by 44
  16. 2022PROD: Progressive Distillation for Dense Retrievalcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 39; 2024: 76; 2025: 35; 2026: 22.

172 papers.

Sort:

By area:

  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  7. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  8. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
  9. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  10. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  11. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  12. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 70
38 more
  1. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  2. 2023Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planningcited by 61
  3. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  4. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  5. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  6. 2023Towards LLM-guided Causal Explainability for Black-box Text Classifierscited by 44
  7. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 41
  8. 2023Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?cited by 39
  9. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 39
  10. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  11. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  12. 2023Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instructioncited by 37
  13. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  14. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  15. 2023Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomiescited by 34
  16. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
  17. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  18. 2024Planning Like Human: A Dual-process Framework for Dialogue Planningcited by 32
  19. 2023Interactive Natural Language Processingcited by 31
  20. 2023Interactive Robot Learning from Verbal Correctioncited by 29
  21. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  22. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  23. 2023Compositional preference models for aligning LMscited by 26
  24. 2024TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervisioncited by 25
  25. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  26. 2024Hate Personified: Investigating the role of LLMs in content moderationcited by 25
  27. 2024Annotation alignment: Comparing LLM and human annotations of conversational safetycited by 23
  28. 2024Performance-Aligned LLMs for Generating Fast Codecited by 21
  29. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  30. 2023Stance Detection: A Practical Guide to Classifying Political Beliefs in Textcited by 20
  31. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  32. 2024Large Language Models as Evaluators for Recommendation Explanationscited by 19
  33. 2024Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiencycited by 18
  34. 2023Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreementcited by 17
  35. 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 17
  36. 2023Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparationcited by 16
  37. 2024Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationalescited by 16
  38. 2024Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotationcited by 16

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Selective Annotation Makes Language Models Better Few-Shot Learners
  2. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling
  3. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answering
  4. 2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation System
  5. 2025Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inference
  6. 2021Generate, Annotate, and Learn: NLP with Synthetic Text
  7. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
  8. 2024Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies