# DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

arXiv: https://arxiv.org/abs/2402.10379 · HTML: https://mlsanity.com/paper/2402.10379/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)

## Built on by (most cited first, 30 of 43)

- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation](https://mlsanity.com/paper/2502.14846/) (2025)
- [Large Language Models Can Self-Improve At Web Agent Tasks](https://mlsanity.com/paper/2405.20309/) (2024)
- [Language Self-Play For Data-Free Training](https://mlsanity.com/paper/2509.07414/) (2025)
- [AI-Assisted Generation of Difficult Math Questions](https://mlsanity.com/paper/2407.21009/) (2024)
- [StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples](https://mlsanity.com/paper/2410.12757/) (2024)
- [SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning](https://mlsanity.com/paper/2407.12874/) (2024)
- [Is Programming by Example solved by LLMs?](https://mlsanity.com/paper/2406.08316/) (2024)
- [Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls](https://mlsanity.com/paper/2510.01631/) (2025)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective](https://mlsanity.com/paper/2410.01720/) (2024)
- [MIND: Math Informed syNthetic Dialogues for Pretraining LLMs](https://mlsanity.com/paper/2410.12881/) (2024)
- [On Pre-training of Multimodal Language Models Customized for Chart Understanding](https://mlsanity.com/paper/2407.14506/) (2024)
- [LLMs can be easily Confused by Instructional Distractions](https://mlsanity.com/paper/2502.04362/) (2025)
- [LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents](https://mlsanity.com/paper/2411.14962/) (2024)
- [Beyond Translation: LLM-Based Data Generation for Multilingual Fact-Checking](https://mlsanity.com/paper/2502.15419/) (2025)
- [On Protecting Agentic Systems' Intellectual Property via Watermarking](https://mlsanity.com/paper/2602.08401/) (2026)
- [SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains](https://mlsanity.com/paper/2507.07229/) (2025)
- [Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning](https://mlsanity.com/paper/2508.21589/) (2025)
- [Exploring Empty Spaces: Human-in-the-Loop Data Augmentation](https://mlsanity.com/paper/2410.01088/) (2024)
- [SS-GEN: A Social Story Generation Framework with Large Language Models](https://mlsanity.com/paper/2406.15695/) (2024)
- [Improving Text Generation on Images with Synthetic Captions](https://mlsanity.com/paper/2406.00505/) (2024)
- [Training Task Experts through Retrieval Based Distillation](https://mlsanity.com/paper/2407.05463/) (2024)
- [Polypersona: Persona-Grounded LLM for Synthetic Survey Responses](https://mlsanity.com/paper/2512.14562/) (2025)
- [Chart-CoCa: Self-Improving Chart Understanding of Vision LMs via Code-Driven Synthesis and Candidate-Conditioned Answering](https://mlsanity.com/paper/2508.11975/) (2025)
- [LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data](https://mlsanity.com/paper/2510.09007/) (2025)
- [mStyleDistance: Multilingual Style Embeddings and their Evaluation](https://mlsanity.com/paper/2502.15168/) (2025)
- [High-Fidelity And Complex Test Data Generation For Google SQL Code Generation Services](https://mlsanity.com/paper/2504.17203/) (2025)
- [An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding](https://mlsanity.com/paper/2505.01743/) (2025)
- [Multi-Model Synthetic Training for Mission-Critical Small Language Models](https://mlsanity.com/paper/2509.13047/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have become a dominant and important tool for NLP researchers in a wide range of tasks. Today, many researchers use LLMs in synthetic data generation, task evaluation, fine-tuning, distillation, and other model-in-the-loop research workflows. However, challenges arise when using these models that stem from their scale, their closed source nature, and the lack of standardized tooling for these new and emerging workflows. The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them. In this paper, we introduce DataDreamer, an open source Python library that allows researchers to write simple code to implement powerful LLM workflows. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science and reproducibility. The library and documentation are available at https://github.com/datadreamer-dev/DataDreamer .