mlsanityData to 7 Oct 2026

Generative AI for Synthetic Data Generation: Methods, Challenges and the Future

arXiv 2403.04190 ↗First posted 7 Mar 2024cs.LG cs.AI cs.CLcited by 31

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The recent surge in research focused on generating synthetic data from large language models (LLMs), especially for scenarios with limited data availability, marks a notable shift in Generative Artificial Intelligence (AI). Their ability to perform comparably to real-world data positions this approach as a compelling solution to low-resource challenges. This paper delves into advanced technologies that leverage these gigantic LLMs for the generation of task-specific training data. We outline methodologies, evaluation techniques, and practical applications, discuss the current limitations, and suggest potential pathways for future research.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 74 of its 93 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

74 of 93 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  13. 2019When Does Label Smoothing Help?cited by 985
  14. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  15. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  16. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  17. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  18. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  19. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  20. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  21. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  22. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  23. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  24. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  25. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  26. 2022Unsupervised Medical Image Translation with Adversarial Diffusion Modelscited by 191
  27. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  28. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  29. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  30. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  31. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  32. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  33. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  34. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  35. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  36. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  37. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 102
  38. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 14; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  2. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  3. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistantscited by 10
  4. 2024Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hackcited by 8
  5. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applicationscited by 7
  6. 2024SocialRec: User Activity Based Post Weighted Dynamic Personalized Post Recommendation System in Social Mediacited by 4
  7. 2025Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applicationscited by 4
  8. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  9. 2024Improving Text Generation on Images with Synthetic Captionscited by 3
  10. 2024EVOLVE: Predicting User Evolution and Network Dynamics in Social Media Using Fine-Tuned GPT-like Modelcited by 3
  11. 2025TimeWak: Temporal Chained-Hashing Watermark for Time Series Datacited by 3
  12. 2025Polypersona: Persona-Grounded LLM for Synthetic Survey Responsescited by 3
19 more
  1. 2026Grounding Synthetic Data Generation With Vision and Language Modelscited by 3
  2. 2025Culturally-Nuanced Story Generation for Reasoning in Low-Resource Languages: The Case of Javanese and Sundanesecited by 2
  3. 2025JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generationcited by 2
  4. 2025Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMscited by 2
  5. 2024SDoH-GPT: Using Large Language Models to Extract Social Determinants of Health (SDoH)cited by 1
  6. 2024Enhancing Natural Language Inference Performance with Knowledge Graph for COVID-19 Automated Fact-Checking in Indonesian Languagecited by 1
  7. 2024Medical Imaging Complexity and its Effects on GAN Performancecited by 1
  8. 2025AutoVDC: Automated Vision Data Cleaning Using Vision-Language Modelscited by 1
  9. 2025Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using LLMscited by 1
  10. 2026METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Modelscited by 1
  11. 2024Synthetic data: How could it be used for infectious disease research?cited by 0
  12. 2024MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Taskscited by 0
  13. 2024The Evolution and Future Perspectives of Artificial Intelligence Generated Contentcited by 0
  14. 2025FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMscited by 0
  15. 2025Synthetic Data Generation for Screen Time and App Usagecited by 0
  16. 2025Data Augmentation Supporting a Conversational Agent Designed for Smoking Cessation Support Groupscited by 0
  17. 2026Kissan-Dost: Bridging the Last Mile in Smallholder Precision Agriculture with Conversational IoTcited by 0
  18. 2026Adversarial Arena: Crowdsourcing Data Generation through Interactive Competitioncited by 0
  19. 2026Hate Speech Detection in Turkish and Arabic: A Comprehensive Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
  3. 2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applications
  4. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  5. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  6. 2024Evaluating Language Models as Synthetic Data Generators
  7. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  8. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems