# Synthetic Data Generation Using Large Language Models: Advances in Text and Code

arXiv: https://arxiv.org/abs/2503.14023 · HTML: https://mlsanity.com/paper/2503.14023/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 54 of its 67 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 54)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)

## Built on by (most cited first, 30 of 61)

- [Self-Questioning Language Models](https://mlsanity.com/paper/2508.03682/) (2025)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [Scaling Low-Resource MT via Synthetic Data Generation with LLMs](https://mlsanity.com/paper/2505.14423/) (2025)
- [Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers](https://mlsanity.com/paper/2509.03059/) (2025)
- [Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents](https://mlsanity.com/paper/2510.19771/) (2025)
- [Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs](https://mlsanity.com/paper/2601.17058/) (2026)
- [Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings](https://mlsanity.com/paper/2509.24506/) (2025)
- [EntropyLong: Effective Long-Context Training via Predictive Uncertainty](https://mlsanity.com/paper/2510.02330/) (2025)
- [Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection](https://mlsanity.com/paper/2601.09692/) (2026)
- [Pre-Trained AI Model Assisted Online Decision-Making under Missing Covariates: A Theoretical Perspective](https://mlsanity.com/paper/2507.07852/) (2025)
- [Filtering with Confidence: When Data Augmentation Meets Conformal Prediction](https://mlsanity.com/paper/2509.21479/) (2025)
- [Privasis: Synthesizing the Largest "Public" Private Dataset from Scratch](https://mlsanity.com/paper/2602.03183/) (2026)
- [Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation](https://mlsanity.com/paper/2602.12172/) (2026)
- [S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data](https://mlsanity.com/paper/2605.01248/) (2026)
- [TF1-EN-3M: Three Million Synthetic Moral Fables for Training Small, Open Language Models](https://mlsanity.com/paper/2504.20605/) (2025)
- [Does Prompt Design Impact Quality of Data Imputation by LLMs?](https://mlsanity.com/paper/2506.04172/) (2025)
- [LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points](https://mlsanity.com/paper/2508.01317/) (2025)
- [Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Era](https://mlsanity.com/paper/2508.19570/) (2025)
- [Building Large-Scale English-Romanian Literary Translation Resources with Open Models](https://mlsanity.com/paper/2509.07829/) (2025)
- [SynQuE: Estimating Synthetic Dataset Quality Without Annotations](https://mlsanity.com/paper/2511.03928/) (2025)
- [BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages](https://mlsanity.com/paper/2511.10338/) (2025)
- [Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://mlsanity.com/paper/2602.05648/) (2026)
- [Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data](https://mlsanity.com/paper/2603.00059/) (2026)
- [Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents](https://mlsanity.com/paper/2603.02239/) (2026)
- [GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning](https://mlsanity.com/paper/2603.10243/) (2026)
- [CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems](https://mlsanity.com/paper/2604.22313/) (2026)
- [Mechanistic Circuit Identification for Controllable Data Generation](https://mlsanity.com/paper/2608.24065/) (2026)
- [EmoGRACE: Aspect-based emotion analysis for social media data](https://mlsanity.com/paper/2503.15133/) (2025)
- [From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning](https://mlsanity.com/paper/2505.14425/) (2025)
- [SQL-Exchange: Transforming SQL Queries Across Domains](https://mlsanity.com/paper/2508.07087/) (2025)

## Abstract (arXiv)

This survey reviews how large language models (LLMs) are transforming synthetic training data generation in both natural language and code domains. By producing artificial but task-relevant examples, these models can significantly augment or even substitute for real-world datasets, particularly in scenarios where labeled data is scarce, expensive, or sensitive. This paper surveys recent advances in leveraging LLMs to create synthetic text and code, highlighting key techniques such as prompt-based generation, retrieval-augmented pipelines, and iterative self-refinement. We examine how these methods can enrich low-resource tasks (e.g., classification, question answering) and facilitate code-centric applications (e.g., instruction tuning, code translation, bug repair) through automated verification of functional correctness. Alongside potential benefits - cost-effectiveness, broad coverage, and controllable diversity - we discuss the accompanying challenges, including factual inaccuracies in generated text, insufficient stylistic or distributional realism, and risks of bias amplification. Proposed mitigation strategies range from filtering and weighting synthetic outputs to reinforcement learning with execution feedback in code domains. We conclude by outlining open research directions, such as automated prompt engineering, cross-modal data synthesis, and robust evaluation frameworks, underscoring the growing importance of LLM-generated synthetic data in accelerating AI development while emphasizing ethical and quality safeguards.