Data to 7 Oct 2026

Data Augmentation using Pre-trained Transformer Models

arXiv 2003.02245 ↗First posted 4 Mar 2020cs.CL cs.LGcited by 168

Not on the map. Why?

Abstract (arXiv)

Language model based pre-trained models such as BERT have provided significant gains across different NLP tasks. In this paper, we study different types of transformer based pre-trained models such as auto-regressive models (GPT-2), auto-encoder models (BERT), and seq2seq models (BART) for conditional data augmentation. We show that prepending the class labels to text sequences provides a simple yet effective way to condition the pre-trained models for data augmentation. Additionally, on three classification benchmarks, pre-trained Seq2Seq model outperforms other data augmentation methods in a low-resource setting. Further, we explore how different pre-trained model based data augmentation differs in-terms of data diversity, and how well such methods preserve the class-label information.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 43; 2022: 43; 2023: 47; 2024: 25; 2025: 7; 2026: 3.

168 papers.

Sort:

By area:

2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
2023Improving CLIP Training with Language Rewritescited by 216
2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
2021Generating Datasets with Pretrained Language Modelscited by 192
2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121

38 more

2021A Survey on Data Augmentation for Text Classificationcited by 111
2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
2021Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Testscited by 77
2021Towards Zero-Label Language Learningcited by 76
2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
2021AEDA: An Easier Data Augmentation Technique for Text Classificationcited by 67
2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
2022Revisiting DocRED -- Addressing the False Negative Problem in Relation Extractioncited by 57
2024TinyAgent: Function Calling at the Edgecited by 55
2022PromDA: Prompt-based Data Augmentation for Low-Resource NLU Taskscited by 51
2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
2021Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarizationcited by 46
2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
2021MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NERcited by 42
2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
2024A Synthetic Dataset for Personal Attribute Inferencecited by 35
2021Gradient Imitation Reinforcement Learning for Low Resource Relation Extractioncited by 33
2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
2021PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decodingcited by 32
2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
2023ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPTcited by 32
2024A Survey on Data Augmentation in Large Model Eracited by 31
2023Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matchingcited by 30
2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29
2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
2021Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifierscited by 24

50 most cited of 168 All 168 (JSON)