# Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges

arXiv: https://arxiv.org/abs/2403.02990 · HTML: https://mlsanity.com/paper/2403.02990/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 88 of its 152 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 30 of 88)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [A Survey on Knowledge Graph-Based Recommender Systems](https://mlsanity.com/paper/2003.00911/) (2020)

## Built on by (most cited first, 30 of 30)

- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators](https://mlsanity.com/paper/2512.19682/) (2025)
- [InternalInspector I2: Robust Confidence Estimation in LLMs through Internal States](https://mlsanity.com/paper/2406.12053/) (2024)
- [Content Moderation by LLM: From Accuracy to Legitimacy](https://mlsanity.com/paper/2409.03219/) (2024)
- [Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data](https://mlsanity.com/paper/2409.11423/) (2024)
- [Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization](https://mlsanity.com/paper/2502.05605/) (2025)
- [Scaling Low-Resource MT via Synthetic Data Generation with LLMs](https://mlsanity.com/paper/2505.14423/) (2025)
- [Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework](https://mlsanity.com/paper/2502.18049/) (2025)
- [From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs](https://mlsanity.com/paper/2601.03808/) (2026)
- [ToxiLab: How Well Do Open-Source LLMs Generate Synthetic Toxicity Data?](https://mlsanity.com/paper/2411.15175/) (2024)
- [Labeling Free-text Data using Language Model Ensembles](https://mlsanity.com/paper/2501.08413/) (2025)
- [Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?](https://mlsanity.com/paper/2410.14602/) (2024)
- [More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning](https://mlsanity.com/paper/2510.07169/) (2025)
- [LlamBERT: Large-scale low-cost data annotation in NLP](https://mlsanity.com/paper/2403.15938/) (2024)
- [Towards Ontology-Enhanced Representation Learning for Large Language Models](https://mlsanity.com/paper/2405.20527/) (2024)
- [Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation?](https://mlsanity.com/paper/2407.02062/) (2024)
- [Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model](https://mlsanity.com/paper/2409.07088/) (2024)
- [Filtering with Confidence: When Data Augmentation Meets Conformal Prediction](https://mlsanity.com/paper/2509.21479/) (2025)
- [Lifelong Event Detection with Embedding Space Separation and Compaction](https://mlsanity.com/paper/2404.02507/) (2024)
- [X-ray2CTPA: Leveraging Diffusion Models to Enhance Pulmonary Embolism Classification](https://mlsanity.com/paper/2406.16109/) (2024)
- [Explaining Matters: Leveraging Definitions and Semantic Expansion for Sexism Detection](https://mlsanity.com/paper/2506.06238/) (2025)
- [LLMSynthor: Macro-Aligned Micro-Records Synthesis with Large Language Models](https://mlsanity.com/paper/2505.14752/) (2025)
- [Towards Bridging Review Sparsity in Recommendation with Textual Edge Graph Representation](https://mlsanity.com/paper/2508.01128/) (2025)
- [LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models](https://mlsanity.com/paper/2602.15675/) (2026)
- [Multi-turn Natural Language to Graph Query Language Translation](https://mlsanity.com/paper/2508.01871/) (2025)
- [Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing](https://mlsanity.com/paper/2509.08329/) (2025)
- [Overcoming the Generalization Limits of SLM Finetuning for Shape-Based Extraction of Datatype and Object Properties](https://mlsanity.com/paper/2511.03407/) (2025)
- [Adaptive Data Augmentation with Multi-armed Bandit: Sample-Efficient Embedding Calibration for Implicit Pattern Recognition](https://mlsanity.com/paper/2602.19385/) (2026)
- [Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention](https://mlsanity.com/paper/2608.26168/) (2026)

## Abstract (arXiv)

In the rapidly evolving field of large language models (LLMs), data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of LLMs on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond. From both data and learning perspectives, we examine various strategies that utilize LLMs for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for diverse forms of further training. Additionally, this paper highlights the primary open challenges faced in this domain, ranging from controllable data augmentation to multi-modal data augmentation. This survey highlights a paradigm shift introduced by LLMs in DA, and aims to serve as a comprehensive guide for researchers and practitioners.