mlsanityData to 7 Oct 2026

A Survey on Data Synthesis and Augmentation for Large Language Models

arXiv 2410.12896 ↗First posted 16 Oct 2024cs.CLcited by 40

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The success of Large Language Models (LLMs) is inherently linked to the availability of vast, diverse, and high-quality data for training and evaluation. However, the growth rate of high-quality data is significantly outpaced by the expansion of training datasets, leading to a looming data exhaustion crisis. This underscores the urgent need to enhance data efficiency and explore new data sources. In this context, synthetic data has emerged as a promising solution. Currently, data generation primarily consists of two major approaches: data augmentation and synthesis. This paper comprehensively reviews and summarizes data generation techniques throughout the lifecycle of LLMs, including data preparation, pre-training, fine-tuning, instruction-tuning, preference alignment, and applications. Furthermore, We discuss the current constraints faced by these methods and investigate potential pathways for future development and research. Our aspiration is to equip researchers with a clear understanding of these methodologies, enabling them to swiftly identify appropriate data generation strategies in the construction of LLMs, while providing valuable insights for future exploration.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 145 of its 238 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

145 of 238 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personas28 Jun
  2. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasets22 Apr
  3. 2024Best Practices and Lessons Learned on Synthetic Data11 Apr

Other references · 142

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2015Variational Inference with Normalizing Flowscited by 2,161
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  13. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  14. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  15. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  16. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  17. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  18. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  19. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  20. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  21. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  22. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  23. 2023StarCoder: may the source be with you!cited by 790
  24. 2022Galactica: A Large Language Model for Sciencecited by 736
  25. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  26. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  27. 2022A Survey on In-context Learningcited by 717
  28. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  29. 2022Large Language Models Can Self-Improvecited by 656
  30. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  31. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  32. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  33. 2023Textbooks Are All You Needcited by 561
  34. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  35. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  36. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  37. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  38. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472

50 most cited of 142 All 142 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 27; 2026: 12.

40 papers.

Sort:

By area:

  1. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  3. 2025OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modelingcited by 48
  4. 2025Self-Questioning Language Modelscited by 47
  5. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 23
  6. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  7. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  8. 2025HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devicescited by 15
  9. 2025DeepPersona: A Generative Engine for Scaling Deep Synthetic Personascited by 14
  10. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agentscited by 13
  11. 2025Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderationcited by 12
  12. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
28 more
  1. 2025OpenRSD: Towards Open-prompts for Object Detection in Remote Sensing Imagescited by 9
  2. 2026Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMscited by 9
  3. 2024LightOcc: Lightweight Spatial Embedding for Efficient Vision-based 3D Occupancy Predictioncited by 6
  4. 2025TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioningcited by 6
  5. 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agentscited by 6
  6. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  7. 2025DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoningcited by 6
  8. 2025SeriesBench: A Benchmark for Narrative-Driven Drama Series Understandingcited by 5
  9. 2025Explainable AI: XAI-Guided Context-Aware Data Augmentationcited by 5
  10. 2025Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Reportcited by 5
  11. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  12. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoningcited by 4
  13. 2026Learning to Disprove: Formal Counterexample Generation with Large Language Modelscited by 4
  14. 2025Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?cited by 3
  15. 2025SkeletonX: Data-Efficient Skeleton-based Action Recognition via Cross-sample Feature Aggregationcited by 3
  16. 2026How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learningcited by 3
  17. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shiftscited by 2
  18. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  19. 2025A Data Synthesis Method Driven by Large Language Models for Proactive Mining of Implicit User Intentions in Tourismcited by 1
  20. 2025NeuroGen: Neural Network Parameter Generation via Large Language Modelscited by 1
  21. 2026MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learningcited by 1
  22. 2026Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Designcited by 1
  23. 2025Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understandingcited by 0
  24. 2026Adaptive Data Augmentation with Multi-armed Bandit: Sample-Efficient Embedding Calibration for Implicit Pattern Recognitioncited by 0
  25. 2026Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studiescited by 0
  26. 2026A Benchmark for LLM's Understanding of Middle School and High School Science Topicscited by 0
  27. 2026Invent a Dataset: Measuring dataset generation abilities with zero seedcited by 0
  28. 2026Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
  2. 2025Scaling Laws of Synthetic Data for Language Models
  3. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
  4. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  5. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges
  6. 2024Evaluating Language Models as Synthetic Data Generators
  7. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
  8. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls