mlsanityData to 14 Oct 2026

Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis

arXiv 2306.07664 ↗First posted 13 Jun 2023cs.CL cs.AI cs.LGcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In recent years, language models (LMs) have made remarkable progress in advancing the field of natural language processing (NLP). However, the impact of data augmentation (DA) techniques on the fine-tuning (FT) performance of these LMs has been a topic of ongoing debate. In this study, we evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks, including classification and regression types, covering single-sentence and sentence-pair tasks. Contrary to prior assumptions that DA does not contribute to the enhancement of LMs' FT performance, our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. In the most favourable case, continued pre-training improves the performance of FT by more than 10% in the few-shot learning setting. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 18 of its 29 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

18 of 29 references.

References · 18

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  7. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  8. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  9. 2020How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?cited by 41
  10. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
  11. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  12. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
6 more
  1. 2021Extending CLIP for Category-to-image Retrieval in E-commercecited by 24
  2. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  3. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  4. 2020Joint Geographical and Temporal Modeling based on Matrix Factorization for Point-of-Interest Recommendationcited by 11
  5. 2023When and What to Ask Through World States and Text Instructions: IGLU NLP Challenge Solutioncited by 5
  6. 2023Self Contrastive Learning for Session-based Recommendationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  2. 2023Towards Contrastive Learning in Music Video Domaincited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges
  2. 2024Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks
  3. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  4. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
  5. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models
  6. 2025LM-mixup: Text Data Augmentation via Language Model based Mixup
  7. 2022Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks
  8. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge