mlsanityData to 7 Oct 2026

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?

arXiv 2010.01764 ↗First posted 5 Oct 2020cs.LG cs.AI stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. In NLP similar results are reported most commonly for low data regimes, non-pretrained models, or situationally for pretrained models. In this paper we ask how effective these techniques really are when applied to pretrained transformers. Using two popular varieties of task-agnostic data augmentation (not tailored to any particular task), Easy Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrichet al., 2015), we conduct a systematic examination of their effects across 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers, including BERT, XLNet, and RoBERTa. We observe a negative result, finding that techniques which previously reported strong improvements for non-pretrained models fail to consistently improve performance for pretrained transformers, even when training data is limited. We hope this empirical analysis helps inform practitioners where data augmentation techniques may confer improvements.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 10; 2023: 14; 2024: 6; 2025: 0; 2026: 3.

41 papers.

Sort:

By area:

  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  2. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  3. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  4. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  5. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  6. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
  7. 2021Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifierscited by 24
  8. 2023Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applicationscited by 21
  9. 2022A Feature-space Multimodal Data Augmentation Technique for Text-video Retrievalcited by 13
  10. 2022Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solverscited by 9
  11. 2024Improving Black-box Robustness with In-Context Rewritingcited by 8
  12. 2021Good-Enough Example Extrapolationcited by 5
29 more
  1. 2022Data Augmentation for Biomedical Factoid Question Answeringcited by 5
  2. 2022AugCSE: Contrastive Sentence Embedding with Diverse Augmentationscited by 5
  3. 2022DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modelingcited by 3
  4. 2022Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligencecited by 3
  5. 2022Named Entity Recognition in Industrial Tables using Tabular Language Modelscited by 3
  6. 2022Human-in-the-Loop Hate Speech Classification in a Multilingual Contextcited by 3
  7. 2023Mitigating Data Scarcity for Large Language Modelscited by 2
  8. 2023Data Augmentation for Neural NLPcited by 2
  9. 2023NLP-LTU at SemEval-2023 Task 10: The Impact of Data Augmentation and Semi-Supervised Learning Techniques on Text Classification Performance on an Imbalanced Datasetcited by 2
  10. 2023Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysiscited by 2
  11. 2023Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Taskscited by 2
  12. 2024DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgerycited by 2
  13. 2024An Experimental Study on Data Augmentation Techniques for Named Entity Recognition on Low-Resource Domainscited by 2
  14. 2022Dictionary-Assisted Supervised Contrastive Learningcited by 1
  15. 2022A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Taskscited by 1
  16. 2023Effectiveness of Data Augmentation for Parameter Efficient Tuning with Limited Datacited by 1
  17. 2023Not Enough Labeled Data? Just Add Semantics: A Data-Efficient Method for Inferring Online Health Textscited by 1
  18. 2023Data Augmentation for Emotion Detection in Small Imbalanced Text Datacited by 1
  19. 2024AutoAugment Is What You Need: Enhancing Rule-based Augmentation Methods in Low-resource Regimescited by 1
  20. 2024A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processingcited by 1
  21. 2021Back Translation Survey for Improving Text Augmentationcited by 0
  22. 2023Exploring Data Augmentation Methods on Social Media Corporacited by 0
  23. 2023Probabilistic Linguistic Knowledge and Token-level Text Augmentationcited by 0
  24. 2023Investigating Masking-based Data Generation in Language Modelscited by 0
  25. 2023Using Weak Supervision and Data Augmentation in Question Answeringcited by 0
  26. 2024Leveraging Large Language Models for Fuzzy String Matching in Political Sciencecited by 0
  27. 2026Leveraging Sentence-oriented Augmentation and Transformer-Based Architecture for Vietnamese-Bahnaric Translationcited by 0
  28. 2026Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probingcited by 0
  29. 2026Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessmentcited by 0