mlsanityData to 7 Oct 2026

DataGen: Unified Synthetic Dataset Generation via Large Language Models

arXiv 2406.18966 ↗First posted 27 Jun 2024cs.CLcited by 22

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large Language Models (LLMs) such as GPT-4 and Llama3 have significantly impacted various fields by enabling high-quality synthetic data generation and reducing dependence on expensive human-generated datasets. Despite this, challenges remain in the areas of generalization, controllability, diversity, and truthfulness within the existing generative frameworks. To address these challenges, this paper presents DataGen, a comprehensive LLM-powered framework designed to produce diverse, accurate, and highly controllable datasets. DataGen is adaptable, supporting all types of text datasets and enhancing the generative process through innovative mechanisms. To augment data diversity, DataGen incorporates an attribute-guided generation module and a group checking feature. For accuracy, it employs a code-based mathematical assessment for label verification alongside a retrieval-augmented generation technique for factual validation. The framework also allows for user-specified constraints, enabling customization of the data generation process to suit particular requirements. Extensive experiments demonstrate the superior quality of data generated by DataGen, and each module within DataGen plays a critical role in this enhancement. Additionally, DataGen is applied in two practical scenarios: benchmarking LLMs and data augmentation. The results indicate that DataGen effectively supports dynamic and evolving benchmarking and that data augmentation improves LLM capabilities in various domains, including agent-oriented abilities and reasoning skills.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 56 of its 94 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

56 of 94 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasets22 Apr

Other references · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
38 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2023Large Language Modelscited by 774
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  9. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  10. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  11. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  12. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  13. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  14. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  15. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  16. 2021Generating Datasets with Pretrained Language Modelscited by 192
  17. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  18. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  19. 2023Self-Alignment with Instruction Backtranslationcited by 149
  20. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  21. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  22. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  23. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  24. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  25. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  26. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  27. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  28. 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Basescited by 76
  29. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  30. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 73
  31. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  32. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  33. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  34. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  35. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
  36. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  37. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  38. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 9.

22 papers.

Sort:

By area:

  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  3. 2024FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Modelscited by 33
  4. 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 29
  5. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  6. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  7. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  8. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 12
  9. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluationcited by 7
  10. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 7
  11. 2025Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Basecited by 6
  12. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspectivecited by 4
10 more
  1. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructionscited by 3
  2. 2025Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinementcited by 3
  3. 2025The Synthetic Mirror -- Synthetic Data at the Age of Agentic AIcited by 0
  4. 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?cited by 0
  5. 2026Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysiscited by 0
  6. 2026Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learningcited by 0
  7. 2026Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documentscited by 0
  8. 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMscited by 0
  9. 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilitiescited by 0
  10. 2026Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
  3. 2023TarGEN: Targeted Data Generation with Large Language Models
  4. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  6. 2026Efficient Embedding-based Synthetic Data Generation for Complex Reasoning Tasks
  7. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  8. 2025FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs