# Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities

arXiv: https://arxiv.org/abs/2501.18845 · HTML: https://mlsanity.com/paper/2501.18845/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Builds on (most cited first, 27 of 27)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks](https://mlsanity.com/paper/2410.20266/) (2024)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [GenAug: Data Augmentation for Finetuning Text Generators](https://mlsanity.com/paper/2010.01794/) (2020)
- [IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues](https://mlsanity.com/paper/2405.13021/) (2024)
- [PolicyQA: A Reading Comprehension Dataset for Privacy Policies](https://mlsanity.com/paper/2010.02557/) (2020)
- [AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models](https://mlsanity.com/paper/2102.05126/) (2021)
- [The TechQA Dataset](https://mlsanity.com/paper/1911.02984/) (2019)
- [PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation](https://mlsanity.com/paper/2310.14192/) (2023)
- [Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking](https://mlsanity.com/paper/2108.10510/) (2021)
- [Cheap and Good? Simple and Effective Data Augmentation for Low Resource Machine Reading](https://mlsanity.com/paper/2106.04134/) (2021)
- [Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering](https://mlsanity.com/paper/2309.06358/) (2023)
- [ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer](https://mlsanity.com/paper/2309.08583/) (2023)
- [XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs](https://mlsanity.com/paper/2311.08614/) (2023)

## Built on by (most cited first, 22 of 22)

- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [Debunk and Infer: Multimodal Fake News Detection via Diffusion-Generated Evidence and LLM Reasoning](https://mlsanity.com/paper/2506.21557/) (2025)
- [Empowering Tabular Data Preparation with Language Models: Why and How?](https://mlsanity.com/paper/2508.01556/) (2025)
- [Filtering with Confidence: When Data Augmentation Meets Conformal Prediction](https://mlsanity.com/paper/2509.21479/) (2025)
- [LM-mixup: Text Data Augmentation via Language Model based Mixup](https://mlsanity.com/paper/2510.20449/) (2025)
- [Backtranslation and paraphrasing in the LLM era? Comparing data augmentation methods for emotion classification](https://mlsanity.com/paper/2507.14590/) (2025)
- [Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers](https://mlsanity.com/paper/2609.11117/) (2026)
- [CIEGAD: Cluster-Conditioned Interpolative and Extrapolative Framework for Geometry-Aware and Domain-Aligned Data Augmentation](https://mlsanity.com/paper/2512.10178/) (2025)
- [Fidelity, Diversity, and Privacy: A Multi-Dimensional LLM Evaluation for Clinical Data Augmentation](https://mlsanity.com/paper/2604.27014/) (2026)
- [Semantic-preserved Augmentation with Confidence-weighted Fine-tuning for Aspect Category Sentiment Analysis](https://mlsanity.com/paper/2506.07148/) (2025)
- [Transforming Chatbot Text: A Sequence-to-Sequence Approach](https://mlsanity.com/paper/2506.12843/) (2025)
- [Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration](https://mlsanity.com/paper/2509.21530/) (2025)
- [Emotion-Enhanced Multi-Task Learning with LLMs for Aspect Category Sentiment Analysis](https://mlsanity.com/paper/2511.19122/) (2025)
- [Towards Privacy-Preserving Mental Health Support with Large Language Models](https://mlsanity.com/paper/2601.01993/) (2026)
- [Training Language Models with homotokens Leads to Delayed Overfitting](https://mlsanity.com/paper/2601.02867/) (2026)
- [BRIDGE the Gap: Mitigating Bias Amplification in Automated Scoring of English Language Learners via Inter-group Data Augmentation](https://mlsanity.com/paper/2602.23580/) (2026)
- [Improving Latent Generalization Using Test-time Compute](https://mlsanity.com/paper/2604.01430/) (2026)
- [Task-Routed Mixture-of-Experts with Cognitive Appraisal for Implicit Sentiment Analysis](https://mlsanity.com/paper/2605.20916/) (2026)
- [Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute](https://mlsanity.com/paper/2608.09351/) (2026)
- [Class-Structure Preservation Beats Diversity: A Comprehensive Benchmark of Text Augmentation Methods for Imbalanced Text Classification](https://mlsanity.com/paper/2608.12340/) (2026)
- [Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention](https://mlsanity.com/paper/2608.26168/) (2026)
- [Generating the Wild: Individual-Consistent Image-to-Video Generation for Wildlife](https://mlsanity.com/paper/2610.05587/) (2026)

## Abstract (arXiv)

The increasing size and complexity of pre-trained language models have demonstrated superior performance in many applications, but they usually require large training datasets to be adequately trained. Insufficient training sets could unexpectedly make the model overfit and fail to cope with complex tasks. Large language models (LLMs) trained on extensive corpora have prominent text generation capabilities, which improve the quality and quantity of data and play a crucial role in data augmentation. Specifically, distinctive prompt templates are given in personalised tasks to guide LLMs in generating the required content. Recent promising retrieval-based techniques further improve the expressive performance of LLMs in data augmentation by introducing external knowledge to enable them to produce more grounded-truth data. This survey provides an in-depth analysis of data augmentation in LLMs, classifying the techniques into Simple Augmentation, Prompt-based Augmentation, Retrieval-based Augmentation and Hybrid Augmentation. We summarise the post-processing approaches in data augmentation, which contributes significantly to refining the augmented data and enabling the model to filter out unfaithful content. Then, we provide the common tasks and evaluation metrics. Finally, we introduce existing challenges and future opportunities that could bring further improvement to data augmentation.