mlsanityData to 7 Oct 2026

Synthetic Data Generation Using Large Language Models: Advances in Text and Code

arXiv 2503.14023 ↗First posted 18 Mar 2025cs.CLcited by 61

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

This survey reviews how large language models (LLMs) are transforming synthetic training data generation in both natural language and code domains. By producing artificial but task-relevant examples, these models can significantly augment or even substitute for real-world datasets, particularly in scenarios where labeled data is scarce, expensive, or sensitive. This paper surveys recent advances in leveraging LLMs to create synthetic text and code, highlighting key techniques such as prompt-based generation, retrieval-augmented pipelines, and iterative self-refinement. We examine how these methods can enrich low-resource tasks (e.g., classification, question answering) and facilitate code-centric applications (e.g., instruction tuning, code translation, bug repair) through automated verification of functional correctness. Alongside potential benefits - cost-effectiveness, broad coverage, and controllable diversity - we discuss the accompanying challenges, including factual inaccuracies in generated text, insufficient stylistic or distributional realism, and risks of bias amplification. Proposed mitigation strategies range from filtering and weighting synthetic outputs to reinforcement learning with execution feedback in code domains. We conclude by outlining open research directions, such as automated prompt engineering, cross-modal data synthesis, and robust evaluation frameworks, underscoring the growing importance of LLM-generated synthetic data in accelerating AI development while emphasizing ethical and quality safeguards.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 54 of its 67 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

54 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Models16 Oct
  2. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Models27 Jun
  3. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun
  4. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows16 Feb
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations11 Oct

Other references · 49

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
37 more
  1. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  2. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  7. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  8. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  9. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  10. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  11. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  12. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  13. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  14. 2022Is GPT-3 a Good Data Annotator?cited by 208
  15. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  16. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  17. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  18. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  19. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  20. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  21. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  22. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  23. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  24. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  25. 2023GSURE-Based Diffusion Model Training with Corrupted Datacited by 40
  26. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  27. 2024RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Modelscited by 27
  28. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  29. 2022AixBench: A Code Generation Benchmark Datasetcited by 22
  30. 2025Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunitiescited by 21
  31. 2022Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contextscited by 12
  32. 2024Data Generation Using Large Language Models for Text Classification: An Empirical Case Studycited by 11
  33. 2024Training Language Models on Synthetic Edit Sequences Improves Code Synthesiscited by 7
  34. 2024LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Educationcited by 7
  35. 2023Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understandingcited by 4
  36. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  37. 2022A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answeringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 37.

61 papers.

Sort:

By area:

  1. 2025Self-Questioning Language Modelscited by 47
  2. 2026Training Language Models via Neural Cellular Automatacited by 15
  3. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  4. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifierscited by 14
  5. 2025Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agentscited by 9
  6. 2026Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMscited by 9
  7. 2025Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settingscited by 5
  8. 2025EntropyLong: Effective Long-Context Training via Predictive Uncertaintycited by 5
  9. 2026Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selectioncited by 4
  10. 2025Pre-Trained AI Model Assisted Online Decision-Making under Missing Covariates: A Theoretical Perspectivecited by 3
  11. 2026Privasis: Synthesizing the Largest "Public" Private Dataset from Scratchcited by 3
  12. 2025TF1-EN-3M: Three Million Synthetic Moral Fables for Training Small, Open Language Modelscited by 2
38 more
  1. 2025Does Prompt Design Impact Quality of Data Imputation by LLMs?cited by 2
  2. 2025LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Pointscited by 2
  3. 2025Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Eracited by 2
  4. 2025Building Large-Scale English-Romanian Literary Translation Resources with Open Modelscited by 2
  5. 2025Filtering with Confidence: When Data Augmentation Meets Conformal Predictioncited by 2
  6. 2025BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languagescited by 2
  7. 2026Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrewcited by 2
  8. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 2
  9. 2026Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Datacited by 2
  10. 2026Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agentscited by 2
  11. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  12. 2026CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systemscited by 2
  13. 2026S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Datacited by 2
  14. 2025EmoGRACE: Aspect-based emotion analysis for social media datacited by 1
  15. 2025From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoningcited by 1
  16. 2025SQL-Exchange: Transforming SQL Queries Across Domainscited by 1
  17. 2025Verification Limits Code LLM Trainingcited by 1
  18. 2025EvoSyn: Generalizable Evolutionary Data Synthesis for Verifiable Learningcited by 1
  19. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  20. 2025ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Testscited by 1
  21. 2025SynQuE: Estimating Synthetic Dataset Quality Without Annotationscited by 1
  22. 2026CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrievalcited by 1
  23. 2026Unseen-Codebases-Domain Data Synthesis and Training Based on Code Graphscited by 1
  24. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  25. 2026When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selectioncited by 1
  26. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  27. 2026Mechanistic Circuit Identification for Controllable Data Generationcited by 1
  28. 2025APIDA-Chat: Structured Synthesis of API Search Dialogues to Bootstrap Conversational Agentscited by 0
  29. 2025CARPAS: Towards Content-Aware Refinement of Provided Aspects for Summarization in Large Language Modelscited by 0
  30. 2026The Critical Role of Aspects in Measuring Document Similaritycited by 0
  31. 2026TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfictioncited by 0
  32. 2026Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generationcited by 0
  33. 2026Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priorscited by 0
  34. 2026On the use of LLMs to generate a dataset of Neural Networkscited by 0
  35. 2026Asking Forever: Universal Activations Behind Turn Amplification in Conversational LLMscited by 0
  36. 2026Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interactioncited by 0
  37. 2026Design and evaluation of an agentic workflow for crisis-related synthetic tweet datasetscited by 0
  38. 2026Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italiancited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  2. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Future
  3. 2024Evaluating Language Models as Synthetic Data Generators
  4. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
  5. 2025Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
  6. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  7. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  8. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation