# How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?

arXiv: https://arxiv.org/abs/2010.01764 · HTML: https://mlsanity.com/paper/2010.01764/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 30 of 41)

- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2108.06332/) (2021)
- [Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifiers](https://mlsanity.com/paper/2103.14453/) (2021)
- [Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications](https://mlsanity.com/paper/2306.11892/) (2023)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers](https://mlsanity.com/paper/2205.00177/) (2022)
- [Improving Black-box Robustness with In-Context Rewriting](https://mlsanity.com/paper/2402.08225/) (2024)
- [Good-Enough Example Extrapolation](https://mlsanity.com/paper/2109.05602/) (2021)
- [Data Augmentation for Biomedical Factoid Question Answering](https://mlsanity.com/paper/2204.04711/) (2022)
- [AugCSE: Contrastive Sentence Embedding with Diverse Augmentations](https://mlsanity.com/paper/2210.13749/) (2022)
- [Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence](https://mlsanity.com/paper/2207.11076/) (2022)
- [DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modeling](https://mlsanity.com/paper/2204.07679/) (2022)
- [Named Entity Recognition in Industrial Tables using Tabular Language Models](https://mlsanity.com/paper/2209.14812/) (2022)
- [Human-in-the-Loop Hate Speech Classification in a Multilingual Context](https://mlsanity.com/paper/2212.02108/) (2022)
- [Mitigating Data Scarcity for Large Language Models](https://mlsanity.com/paper/2302.01806/) (2023)
- [Data Augmentation for Neural NLP](https://mlsanity.com/paper/2302.11412/) (2023)
- [NLP-LTU at SemEval-2023 Task 10: The Impact of Data Augmentation and Semi-Supervised Learning Techniques on Text Classification Performance on an Imbalanced Dataset](https://mlsanity.com/paper/2304.12847/) (2023)
- [Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis](https://mlsanity.com/paper/2306.07664/) (2023)
- [Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Tasks](https://mlsanity.com/paper/2307.16609/) (2023)
- [DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2401.09239/) (2024)
- [An Experimental Study on Data Augmentation Techniques for Named Entity Recognition on Low-Resource Domains](https://mlsanity.com/paper/2411.14551/) (2024)
- [Dictionary-Assisted Supervised Contrastive Learning](https://mlsanity.com/paper/2210.15172/) (2022)
- [A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks](https://mlsanity.com/paper/2212.05251/) (2022)
- [Effectiveness of Data Augmentation for Parameter Efficient Tuning with Limited Data](https://mlsanity.com/paper/2303.02577/) (2023)
- [Not Enough Labeled Data? Just Add Semantics: A Data-Efficient Method for Inferring Online Health Texts](https://mlsanity.com/paper/2309.09877/) (2023)
- [Data Augmentation for Emotion Detection in Small Imbalanced Text Data](https://mlsanity.com/paper/2310.17015/) (2023)

## Abstract (arXiv)

Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. In NLP similar results are reported most commonly for low data regimes, non-pretrained models, or situationally for pretrained models. In this paper we ask how effective these techniques really are when applied to pretrained transformers. Using two popular varieties of task-agnostic data augmentation (not tailored to any particular task), Easy Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrichet al., 2015), we conduct a systematic examination of their effects across 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers, including BERT, XLNet, and RoBERTa. We observe a negative result, finding that techniques which previously reported strong improvements for non-pretrained models fail to consistently improve performance for pretrained transformers, even when training data is limited. We hope this empirical analysis helps inform practitioners where data augmentation techniques may confer improvements.