# Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis

arXiv: https://arxiv.org/abs/2306.07664 · HTML: https://mlsanity.com/paper/2306.07664/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?](https://mlsanity.com/paper/2010.01764/) (2020)
- [FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2108.06332/) (2021)
- [AdaPrompt: Adaptive Model Training for Prompt-based NLP](https://mlsanity.com/paper/2202.04824/) (2022)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Extending CLIP for Category-to-image Retrieval in E-commerce](https://mlsanity.com/paper/2112.11294/) (2021)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Joint Geographical and Temporal Modeling based on Matrix Factorization for Point-of-Interest Recommendation](https://mlsanity.com/paper/2001.08961/) (2020)
- [When and What to Ask Through World States and Text Instructions: IGLU NLP Challenge Solution](https://mlsanity.com/paper/2305.05754/) (2023)
- [Self Contrastive Learning for Session-based Recommendation](https://mlsanity.com/paper/2306.01266/) (2023)

## Built on by (most cited first, 2 of 2)

- [DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2309.05173/) (2023)
- [Towards Contrastive Learning in Music Video Domain](https://mlsanity.com/paper/2309.00347/) (2023)

## Abstract (arXiv)

In recent years, language models (LMs) have made remarkable progress in advancing the field of natural language processing (NLP). However, the impact of data augmentation (DA) techniques on the fine-tuning (FT) performance of these LMs has been a topic of ongoing debate. In this study, we evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks, including classification and regression types, covering single-sentence and sentence-pair tasks. Contrary to prior assumptions that DA does not contribute to the enhancement of LMs' FT performance, our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. In the most favourable case, continued pre-training improves the performance of FT by more than 10% in the few-shot learning setting. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.