mlsanityData to 7 Oct 2026

Want To Reduce Labeling Cost? GPT-3 Can Help

arXiv 2108.13487 ↗First posted 30 Aug 2021cs.CL cs.AIcited by 196

Not on the map. Why?

Abstract (arXiv)

Data annotation is a time-consuming and labor-intensive process for many NLP tasks. Although there exist various methods to produce pseudo data labels, they are often task-specific and require a decent amount of labeled data to start with. Recently, the immense language model GPT-3 with 175 billion parameters has achieved tremendous improvement across many few-shot learning tasks. In this paper, we explore ways to leverage GPT-3 as a low-cost data labeler to train other models. We find that, to make the downstream model achieve the same performance on a variety of NLU and NLG tasks, it costs 50% to 96% less to use labels from GPT-3 than using labels from humans. Furthermore, we propose a novel framework of combining pseudo labels from GPT-3 with human labels, which leads to even better performance with limited labeling budget. These results present a cost-effective data labeling methodology that is generalizable to many practical applications.

Key facts. Cited by 196 papers in the map (data through 7 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 30 Aug 2021.

Builds on

19 of 40 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  12. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
7 more
  1. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  2. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  3. 2018A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processescited by 52
  4. 2020Meta-learning for Few-shot Natural Language Processing: A Surveycited by 42
  5. 2020Adaptive Self-training for Few-shot Neural Sequence Labelingcited by 18
  6. 2019Dual Supervised Learning for Natural Language Understanding and Generationcited by 14
  7. 2020Dual Supervision Framework for Relation Extraction with Distant Supervision and Human Annotationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 21; 2023: 56; 2024: 67; 2025: 29; 2026: 22.

196 papers.

Sort:

By area:

  1. 2022A Survey on In-context Learningcited by 717
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  5. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  8. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  9. 2022Is GPT-3 a Good Data Annotator?cited by 208
  10. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  11. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  12. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
38 more
  1. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  2. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  3. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  4. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  5. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  6. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  7. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  8. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  9. 2023Automated Annotation with Generative AI Requires Validationcited by 75
  10. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  11. 2023Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planningcited by 60
  12. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  13. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  14. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  15. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  16. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  17. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  18. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  19. 2022Controllable Dialogue Simulation with In-Context Learningcited by 42
  20. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  21. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  22. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  23. 2022Uncertainty Estimation for Language Reward Modelscited by 35
  24. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  25. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  26. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  27. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  28. 2023How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?cited by 30
  29. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  30. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  31. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  32. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  33. 2023Making Large Language Models Better Data Creatorscited by 28
  34. 2025A Comprehensive Survey of Synthetic Tabular Data Generationcited by 27
  35. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 26
  36. 2023Embedding Democratic Values into Social Media AIs via Societal Objective Functionscited by 25
  37. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  38. 2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24

50 most cited of 196 All 196 (JSON)