# Synthetic data, real errors: how (not) to publish and use synthetic data

arXiv: https://arxiv.org/abs/2305.09235 · HTML: https://mlsanity.com/paper/2305.09235/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 62 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [FairGAN: Fairness-aware Generative Adversarial Networks](https://mlsanity.com/paper/1805.11202/) (2018)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks](https://mlsanity.com/paper/2110.12884/) (2021)
- [Synthcity: facilitating innovative use cases of synthetic data in different data modalities](https://mlsanity.com/paper/2301.07573/) (2023)
- [Membership Inference Attacks against Synthetic Data through Overfitting Detection](https://mlsanity.com/paper/2302.12580/) (2023)
- [Multi-Agent Diverse Generative Adversarial Networks](https://mlsanity.com/paper/1704.02906/) (2017)
- [Dual Discriminator Generative Adversarial Nets](https://mlsanity.com/paper/1709.03831/) (2017)
- [Uncertainty-Aware Deep Classifiers using Generative Models](https://mlsanity.com/paper/2006.04183/) (2020)
- [AdaGAN: Boosting Generative Models](https://mlsanity.com/paper/1701.02386/) (2017)
- [Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data](https://mlsanity.com/paper/2304.03722/) (2023)
- [GFlowOut: Dropout with Generative Flow Networks](https://mlsanity.com/paper/2210.12928/) (2022)
- [Boosted Generative Models](https://mlsanity.com/paper/1702.08484/) (2017)
- [Conditional Synthetic Data Generation for Robust Machine Learning Applications with Limited Pandemic Data](https://mlsanity.com/paper/2109.06486/) (2021)
- [Uncertainty Quantification with Generative Models](https://mlsanity.com/paper/1910.10046/) (2019)
- [Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators](https://mlsanity.com/paper/1807.11346/) (2018)
- [Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations](https://mlsanity.com/paper/2201.08186/) (2022)

## Built on by (most cited first, 29 of 29)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [On the Constrained Time-Series Generation Problem](https://mlsanity.com/paper/2307.01717/) (2023)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare](https://mlsanity.com/paper/2401.13716/) (2024)
- [TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Models](https://mlsanity.com/paper/2409.16118/) (2024)
- [Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning](https://mlsanity.com/paper/2410.14208/) (2024)
- [Utility Theory of Synthetic Data Generation](https://mlsanity.com/paper/2305.10015/) (2023)
- [Fill In The Gaps: Model Calibration and Generalization with Synthetic Data](https://mlsanity.com/paper/2410.10864/) (2024)
- [Generating Diverse Agricultural Data for Vision-Based Farming Applications](https://mlsanity.com/paper/2403.18351/) (2024)
- [Struct-Bench: A Benchmark for Differentially Private Structured Text Generation](https://mlsanity.com/paper/2509.10696/) (2025)
- [The Real Deal Behind the Artificial Appeal: Inferential Utility of Tabular Synthetic Data](https://mlsanity.com/paper/2312.07837/) (2023)
- [TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsetting](https://mlsanity.com/paper/2406.01805/) (2024)
- [A text-to-tabular approach to generate synthetic patient data using LLMs](https://mlsanity.com/paper/2412.05153/) (2024)
- [Debiasing Synthetic Data Generated by Deep Generative Models](https://mlsanity.com/paper/2411.04216/) (2024)
- [Provably Improving Generalization of Few-Shot Models with Synthetic Data](https://mlsanity.com/paper/2505.24190/) (2025)
- [Harnessing Synthetic Data from Generative AI for Statistical Inference](https://mlsanity.com/paper/2603.05396/) (2026)
- [A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets](https://mlsanity.com/paper/2402.03985/) (2024)
- [TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data](https://mlsanity.com/paper/2607.03926/) (2026)
- [Generative Semi-supervised Learning with Meta-Optimized Synthetic Samples](https://mlsanity.com/paper/2309.16143/) (2023)
- [Private Synthetic Data Meets Ensemble Learning](https://mlsanity.com/paper/2310.09729/) (2023)
- [GAUDA: Generative Adaptive Uncertainty-guided Diffusion-based Augmentation for Surgical Segmentation](https://mlsanity.com/paper/2501.10819/) (2025)
- [Beyond a Single Mode: GAN Ensembles for Diverse Medical Data Generation](https://mlsanity.com/paper/2503.24258/) (2025)
- [Importance of realism in procedurally-generated synthetic images for deep learning: case studies in maize and canola](https://mlsanity.com/paper/2404.05128/) (2024)
- [A Comparative Study of Open-Source Libraries for Synthetic Tabular Data Generation: SDV vs. SynthCity](https://mlsanity.com/paper/2506.17847/) (2025)
- [ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links](https://mlsanity.com/paper/2509.01387/) (2025)
- [flowengineR: A Modular and Extensible Framework for Fair and Reproducible Workflow Design in R](https://mlsanity.com/paper/2511.00079/) (2025)
- [Task-Conditioned Synthetic Data Generation for Improving Machine Learning Performance in Agricultural Prediction Tasks](https://mlsanity.com/paper/2607.09751/) (2026)
- [ProxyGuard: Direct Reliability Inference for Randomized Data Release Mechanisms with Shared Targets](https://mlsanity.com/paper/2608.18643/) (2026)

## Abstract (arXiv)

Generating synthetic data through generative models is gaining interest in the ML community and beyond, promising a future where datasets can be tailored to individual needs. Unfortunately, synthetic data is usually not perfect, resulting in potential errors in downstream tasks. In this work we explore how the generative process affects the downstream ML task. We show that the naive synthetic data approach -- using synthetic data as if it is real -- leads to downstream models and analyses that do not generalize well to real data. As a first step towards better ML in the synthetic data regime, we introduce Deep Generative Ensemble (DGE) -- a framework inspired by Deep Ensembles that aims to implicitly approximate the posterior distribution over the generative process model parameters. DGE improves downstream model training, evaluation, and uncertainty quantification, vastly outperforming the naive approach on average. The largest improvements are achieved for minority classes and low-density regions of the original data, for which the generative uncertainty is largest.