# An Empirical Survey of Data Augmentation for Limited Data Learning in NLP

arXiv: https://arxiv.org/abs/2106.07499 · HTML: https://mlsanity.com/paper/2106.07499/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Builds on 112 of its 170 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 30 of 112)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)

## Built on by (most cited first, 30 of 101)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Y Social: an LLM-powered Social Media Digital Twin](https://mlsanity.com/paper/2408.00818/) (2024)
- [Attacking Open-domain Question Answering by Injecting Misinformation](https://mlsanity.com/paper/2110.07803/) (2021)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptation](https://mlsanity.com/paper/2202.12350/) (2022)
- [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023)
- [Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding](https://mlsanity.com/paper/2210.14169/) (2022)
- [Continual Sequence Generation with Adaptive Compositional Modules](https://mlsanity.com/paper/2203.10652/) (2022)
- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [Prompt Perturbation Consistency Learning for Robust Language Models](https://mlsanity.com/paper/2402.15833/) (2024)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [A Comparative Study on Enhancing Prediction in Social Network Advertisement through Data Augmentation](https://mlsanity.com/paper/2404.13812/) (2024)
- [Parametric Scattering Networks](https://mlsanity.com/paper/2107.09539/) (2021)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification](https://mlsanity.com/paper/2109.00523/) (2021)
- [TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2205.06153/) (2022)
- [Cheetah: Natural Language Generation for 517 African Languages](https://mlsanity.com/paper/2401.01053/) (2024)
- [Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AI](https://mlsanity.com/paper/2407.21523/) (2024)
- [HyperMixer: An MLP-based Low Cost Alternative to Transformers](https://mlsanity.com/paper/2203.03691/) (2022)
- [To Augment or Not to Augment? A Comparative Study on Text Augmentation Techniques for Low-Resource NLP](https://mlsanity.com/paper/2111.09618/) (2021)
- [Ada-Instruct: Adapting Instruction Generators for Complex Reasoning](https://mlsanity.com/paper/2310.04484/) (2023)
- [DaCy: A Unified Framework for Danish NLP](https://mlsanity.com/paper/2107.05295/) (2021)
- [ALP: Data Augmentation using Lexicalized PCFGs for Few-Shot Text Classification](https://mlsanity.com/paper/2112.11916/) (2021)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP](https://mlsanity.com/paper/2404.00415/) (2024)
- [Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation](https://mlsanity.com/paper/2203.11670/) (2022)
- [Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning](https://mlsanity.com/paper/2204.04813/) (2022)

## Abstract (arXiv)

NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets. The dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks where significant time, money, or expertise is required to label massive amounts of textual data. Recently, data augmentation methods have been explored as a means of improving data efficiency in NLP. To date, there has been no systematic empirical overview of data augmentation for NLP in the limited labeled data setting, making it difficult to understand which methods work in which settings. In this paper, we provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting, summarizing the landscape of methods (including token-level augmentations, sentence-level augmentations, adversarial augmentations, and hidden-space augmentations) and carrying out experiments on 11 datasets covering topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the results, we draw several conclusions to help practitioners choose appropriate augmentations in different settings and discuss the current challenges and future directions for limited data learning in NLP.