mlsanityData to 7 Oct 2026

GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation

arXiv 2104.08826 ↗First posted 18 Apr 2021cs.CL cs.AIcited by 185

Not on the map. Why?

Cited by 24 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. Ablation studies and a qualitative analysis provide more insights into our approach.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

40 of 59 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
28 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  8. 2016Sequence-Level Knowledge Distillationcited by 761
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2019Adversarial Training for Free!cited by 638
  11. 2015Generating Sentences from a Continuous Spacecited by 607
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  13. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  14. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  15. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  16. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  17. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  18. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  19. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  20. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  21. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  22. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  23. 2020SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustnesscited by 94
  24. 2020Sequence-Level Mixed Sample Data Augmentationcited by 71
  25. 2018Sequence-to-Sequence Data Augmentation for Dialogue Language Understandingcited by 67
  26. 2020CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detectioncited by 23
  27. 2018Data Augmentation for Spoken Language Understanding via Joint Variational Generationcited by 21
  28. 2020Composed Variational Natural Language Generation for Few-shot Intentscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 28; 2023: 53; 2024: 60; 2025: 27; 2026: 10.

185 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  4. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  5. 2022Large Language Models Are Reasoning Teacherscited by 318
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  7. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  8. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  9. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  10. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  11. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  12. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
38 more
  1. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  2. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  3. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  4. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  5. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  6. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  7. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  8. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  9. 2024TinyAgent: Function Calling at the Edgecited by 55
  10. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  11. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  12. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  13. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  14. 2024A Comprehensive Survey on Data Augmentationcited by 38
  15. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  16. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  17. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  18. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  19. 2024A Synthetic Dataset for Personal Attribute Inferencecited by 35
  20. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  21. 2023Evaluation of ChatGPT Model for Vulnerability Detectioncited by 33
  22. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
  23. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  24. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  25. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  26. 2023How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?cited by 30
  27. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  28. 2022Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generationcited by 28
  29. 2023Making Large Language Models Better Data Creatorscited by 28
  30. 2023COKE: A Cognitive Knowledge Graph for Machine Theory of Mindcited by 26
  31. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25
  32. 2023People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detectioncited by 25
  33. 2023Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Informationcited by 24
  34. 2023Mixture of Soft Prompts for Controllable Data Generationcited by 23
  35. 2024SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluationcited by 23
  36. 2023Using GPT-4 to Augment Unbalanced Data for Automatic Scoringcited by 20
  37. 2023Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Modelscited by 20
  38. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19

50 most cited of 185 All 185 (JSON)