mlsanityData to 7 Oct 2026

On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

arXiv 2406.15126 ↗First posted 14 Jun 2024cs.CLcited by 240

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Within the evolving landscape of deep learning, the dilemma of data quantity and quality has been a long-standing problem. The recent advent of Large Language Models (LLMs) offers a data-centric solution to alleviate the limitations of real-world data with synthetic data generation. However, current investigations into this field lack a unified framework and mostly stay on the surface. Therefore, this paper provides an organization of relevant studies based on a generic workflow of synthetic data generation. By doing so, we highlight the gaps within existing research and outline prospective avenues for future study. This work aims to shepherd the academic and industrial communities towards deeper, more methodical inquiries into the capabilities and applications of LLMs-driven synthetic data generation.

Key facts. Cited by 240 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 78 of its 110 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

78 of 110 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasets22 Apr
  2. 2024Best Practices and Lessons Learned on Synthetic Data11 Apr

Other references · 76

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
38 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  7. 2022Large Language Models Can Self-Improvecited by 656
  8. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  10. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  13. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  14. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  15. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  16. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  17. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  18. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  19. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  20. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  21. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  22. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  23. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  24. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  25. 2022Is GPT-3 a Good Data Annotator?cited by 208
  26. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  27. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  28. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  29. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  30. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  31. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  32. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  33. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  34. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  35. 2024Long-form factuality in large language modelscited by 146
  36. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  37. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  38. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 139; 2026: 58.

240 papers.

Sort:

By area:

  1. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  2. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  3. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  4. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  5. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  6. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  7. 2025Self-Questioning Language Modelscited by 47
  8. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  9. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  10. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  11. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  12. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
38 more
  1. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  2. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  3. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  4. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  5. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  6. 2024Efficacy of Synthetic Data as a Benchmarkcited by 29
  7. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  8. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 25
  9. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  10. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  11. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  12. 2025Foundation Models for Spatio-Temporal Data Science: A Tutorial and Surveycited by 23
  13. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  14. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  15. 2024Can LLMs Convert Graphs to Text-Attributed Graphs?cited by 19
  16. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generationcited by 17
  17. 2025Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimizationcited by 16
  18. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 16
  19. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  20. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  21. 2026Agent-as-a-Judgecited by 15
  22. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  23. 2024HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Datacited by 14
  24. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMscited by 14
  25. 2024Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Studycited by 14
  26. 2025Large Language Models in Argument Mining: A Surveycited by 14
  27. 2025Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Modelscited by 13
  28. 2024Scaling Parameter-Constrained Language Models with Quality Datacited by 12
  29. 2024LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documentscited by 12
  30. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  31. 2025LLMs can be easily Confused by Instructional Distractionscited by 12
  32. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  33. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 11
  34. 2025LLM-Empowered Class Imbalanced Graph Prompt Learning for Online Drug Trafficking Detectioncited by 11
  35. 2024Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-developmentcited by 10
  36. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  37. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  38. 2025RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressionscited by 10

50 most cited of 240 All 240 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Future
  2. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  3. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
  4. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  5. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  6. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  7. 2025FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs
  8. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Models