# Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers

arXiv: https://arxiv.org/abs/2103.14453 · HTML: https://mlsanity.com/paper/2103.14453/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 78 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Stochastic Pooling for Regularization of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1301.3557/) (2013)
- [Data Augmentation via Dependency Tree Morphing for Low-Resource Languages](https://mlsanity.com/paper/1903.09460/) (2019)
- [Learning Data Manipulation for Augmentation and Weighting](https://mlsanity.com/paper/1910.12795/) (2019)
- [How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?](https://mlsanity.com/paper/2010.01764/) (2020)
- [Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs](https://mlsanity.com/paper/1812.04718/) (2018)
- [Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation](https://mlsanity.com/paper/1601.03651/) (2016)

## Built on by (most cited first, 24 of 24)

- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks](https://mlsanity.com/paper/2304.13861/) (2023)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [GDA: Generative Data Augmentation Techniques for Relation Extraction Tasks](https://mlsanity.com/paper/2305.16663/) (2023)
- [Data Augmentation Through Random Style Replacement](https://mlsanity.com/paper/2504.10563/) (2025)
- [Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization](https://mlsanity.com/paper/2305.16259/) (2023)
- [A Review on Text-Based Emotion Detection -- Techniques, Applications, Datasets, and Future Directions](https://mlsanity.com/paper/2205.03235/) (2022)
- [Iterative Data Generation with Large Language Models for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2407.00341/) (2024)
- [Data Augmentation for Biomedical Factoid Question Answering](https://mlsanity.com/paper/2204.04711/) (2022)
- [Explainable AI: XAI-Guided Context-Aware Data Augmentation](https://mlsanity.com/paper/2506.03484/) (2025)
- [Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence](https://mlsanity.com/paper/2207.11076/) (2022)
- [Knowledge-Grounded Conversational Data Augmentation with Generative Conversational Networks](https://mlsanity.com/paper/2207.11363/) (2022)
- [On Evaluation Protocols for Data Augmentation in a Limited Data Scenario](https://mlsanity.com/paper/2402.14895/) (2024)
- [Improving Adversarial Robustness via Decoupled Visual Representation Masking](https://mlsanity.com/paper/2406.10933/) (2024)
- [RISC: Generating Realistic Synthetic Bilingual Insurance Contract](https://mlsanity.com/paper/2304.04212/) (2023)
- [DKE-Research at SemEval-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness](https://mlsanity.com/paper/2404.09206/) (2024)
- [WHERE and WHICH: Iterative Debate for Biomedical Synthetic Data Augmentation](https://mlsanity.com/paper/2503.23673/) (2025)
- [AIGenC: An AI generalisation model via creativity](https://mlsanity.com/paper/2205.09738/) (2022)
- [Use the Detection Transformer as a Data Augmenter](https://mlsanity.com/paper/2304.04554/) (2023)
- [Text Fact Transfer](https://mlsanity.com/paper/2310.14486/) (2023)
- [Advancing Ear Biometrics: Enhancing Accuracy and Robustness through Deep Learning](https://mlsanity.com/paper/2406.00135/) (2024)
- [Combining Autoregressive and Autoencoder Language Models for Text Classification](https://mlsanity.com/paper/2411.13282/) (2024)
- [A Comprehensive Survey of Challenges and Opportunities of Few-Shot Learning Across Multiple Domains](https://mlsanity.com/paper/2504.04017/) (2025)
- [Cross Paraphrastic Invariance Learning for Hallucination Detection](https://mlsanity.com/paper/2606.08157/) (2026)

## Abstract (arXiv)

In many cases of machine learning, research suggests that the development of training data might have a higher relevance than the choice and modelling of classifiers themselves. Thus, data augmentation methods have been developed to improve classifiers by artificially created training data. In NLP, there is the challenge of establishing universal rules for text transformations which provide new linguistic patterns. In this paper, we present and evaluate a text generation method suitable to increase the performance of classifiers for long and short texts. We achieved promising improvements when evaluating short as well as long text tasks with the enhancement by our text generation method. Especially with regard to small data analytics, additive accuracy gains of up to 15.53% and 3.56% are achieved within a constructed low data regime, compared to the no augmentation baseline and another data augmentation technique. As the current track of these constructed regimes is not universally applicable, we also show major improvements in several real world low data tasks (up to +4.84 F1-score). Since we are evaluating the method from many perspectives (in total 11 datasets), we also observe situations where the method might not be suitable. We discuss implications and patterns for the successful application of our approach on different types of datasets.