mlsanityData to 7 Oct 2026

A Survey of Data Synthesis Approaches

arXiv 2407.03672 ↗First posted 4 Jul 2024cs.LG cs.AIcited by 8

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

This paper provides a detailed survey of synthetic data techniques. We first discuss the expected goals of using synthetic data in data augmentation, which can be divided into four parts: 1) Improving Diversity, 2) Data Balancing, 3) Addressing Domain Shift, and 4) Resolving Edge Cases. Synthesizing data are closely related to the prevailing machine learning techniques at the time, therefore, we summarize the domain of synthetic data techniques into four categories: 1) Expert-knowledge, 2) Direct Training, 3) Pre-train then Fine-tune, and 4) Foundation Models without Fine-tuning. Next, we categorize the goals of synthetic data filtering into four types for discussion: 1) Basic Quality, 2) Label Consistency, and 3) Data Distribution. In section 5 of this paper, we also discuss the future directions of synthetic data and state three direction that we believe is important: 1) focus more on quality, 2) the evaluation of synthetic data, and 3) multi-model data augmentation.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 49 of its 77 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

49 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Best Practices and Lessons Learned on Synthetic Data11 Apr

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  11. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  12. 2022A Survey on In-context Learningcited by 717
36 more
  1. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  2. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  3. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  4. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  5. 2017Data Augmentation Generative Adversarial Networkscited by 278
  6. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  7. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  8. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  9. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  10. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  11. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  12. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  13. 2020Training Question Answering Models From Synthetic Datacited by 95
  14. 2023Generating images of rare concepts using pre-trained diffusion modelscited by 78
  15. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  16. 2021AEDA: An Easier Data Augmentation Technique for Text Classificationcited by 67
  17. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  18. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  19. 2020Affinity and Diversity: Quantifying Mechanisms of Data Augmentationcited by 45
  20. 2018Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!cited by 43
  21. 2020Paraphrase Augmented Task-Oriented Dialog Generationcited by 43
  22. 2020GenAug: Data Augmentation for Finetuning Text Generatorscited by 41
  23. 2024Synthetic Dialogue Dataset Generation using LLM Agentscited by 39
  24. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  25. 2017Learning Paraphrastic Sentence Embeddings from Back-Translated Bitextcited by 27
  26. 2016Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentationcited by 26
  27. 2021AUGNLG: Few-shot Natural Language Generation using Self-trained Data Augmentationcited by 17
  28. 2022DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detectioncited by 17
  29. 2022Learning Multimodal Data Augmentation in Feature Spacecited by 17
  30. 2023CamDiff: Camouflage Image Augmentation via Diffusion Modelcited by 16
  31. 2021Data Augmentation for Cross-Domain Named Entity Recognitioncited by 14
  32. 2019Transforming Wikipedia into Augmented Data for Query-Focused Summarizationcited by 12
  33. 2024VIXEN: Visual Text Comparison Network for Image Difference Captioningcited by 12
  34. 2023Semantic Generative Augmentations for Few-Shot Countingcited by 6
  35. 2024Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generationcited by 4
  36. 2022Hands-Up: Leveraging Synthetic Data for Hands-On-Wheel Detectioncited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 5.

8 papers.

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  2. 2024HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Datacited by 14
  3. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  4. 2025TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioningcited by 6
  5. 2025SGD-Mix: Enhancing Domain-Specific Image Classification with Label-Preserving Data Augmentationcited by 3
  6. 2024Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Modelscited by 1
  7. 2025Spatial Knowledge Graph-Guided Multimodal Synthesiscited by 0
  8. 2025Towards High Supervised Learning Utility Training Data Generation: Data Pruning and Column Reorderingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models
  2. 2024Comprehensive Exploration of Synthetic Data Generation: A Survey
  3. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matching
  4. 2023Boosting Data Analytics With Synthetic Volume Expansion
  5. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory
  6. 2025Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Era
  7. 2026Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
  8. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models