mlsanityData to 7 Oct 2026

Is GPT-3 a Good Data Annotator?

arXiv 2212.10450 ↗First posted 20 Dec 2022cs.CLcited by 208

Not on the map. Why?

Abstract (arXiv)

Data annotation is the process of labeling data that could be used to train machine learning models. Having high-quality annotation is crucial, as it allows the model to learn the relationship between the input data and the desired output. GPT-3, a large-scale language model developed by OpenAI, has demonstrated impressive zero- and few-shot performance on a wide range of NLP tasks. It is therefore natural to wonder whether it can be used to effectively annotate data for NLP tasks. In this paper, we evaluate the performance of GPT-3 as a data annotator by comparing it with traditional data annotation methods and analyzing its output on a range of tasks. Through this analysis, we aim to provide insight into the potential of GPT-3 as a general-purpose data annotator in NLP.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Builds on 50 of its 78 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

50 of 78 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  12. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  13. 2022Galactica: A Large Language Model for Sciencecited by 736
  14. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  15. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  16. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  17. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  18. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  19. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  20. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  21. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  22. 2021How Many Data Points is a Prompt Worth?cited by 197
  23. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  24. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  25. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  26. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  27. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 102
  28. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  29. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  30. 2017Adversarial Multi-task Learning for Text Classificationcited by 91
  31. 2020Position-Aware Tagging for Aspect Sentiment Triplet Extractioncited by 90
  32. 2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
  33. 2021Learning Span-Level Interactions for Aspect Sentiment Triplet Extractioncited by 60
  34. 2021Changing the World by Changing the Datacited by 51
  35. 2021GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systemscited by 30
  36. 2019When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classificationcited by 13
  37. 2021Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triplescited by 13
  38. 2022Ask Me What You Need: Product Retrieval using Knowledge from GPT-3cited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 58; 2024: 71; 2025: 51; 2026: 27.

208 papers.

Sort:

By area:

  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  2. 2022A Survey on In-context Learningcited by 717
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  4. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  5. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  10. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  11. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  12. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
38 more
  1. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  2. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  3. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  4. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  5. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  6. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  7. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  8. 2023LLMs Accelerate Annotation for Medical Information Extractioncited by 77
  9. 2023Automated Annotation with Generative AI Requires Validationcited by 75
  10. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  11. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  12. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  13. 2023Is GPT-4 a Good Data Analyst?cited by 69
  14. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  15. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  16. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  17. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patternscited by 54
  18. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  19. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  20. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  21. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  22. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  23. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  24. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  25. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  26. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  27. 2023Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuningcited by 31
  28. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  29. 2023How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?cited by 30
  30. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  31. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  32. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Triggercited by 29
  33. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  34. 2024TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervisioncited by 25
  35. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  36. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 24
  37. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  38. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 23

50 most cited of 208 All 208 (JSON)