# Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data

arXiv: https://arxiv.org/abs/2304.03722 · HTML: https://mlsanity.com/paper/2304.03722/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 92 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 30 of 51)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Reliable Fidelity and Diversity Metrics for Generative Models](https://mlsanity.com/paper/2002.09797/) (2020)
- [Assessing Generative Models via Precision and Recall](https://mlsanity.com/paper/1806.00035/) (2018)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [Coupled Generative Adversarial Networks](https://mlsanity.com/paper/1606.07536/) (2016)

## Built on by (most cited first, 29 of 29)

- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [On the Constrained Time-Series Generation Problem](https://mlsanity.com/paper/2307.01717/) (2023)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Synthetic data, real errors: how (not) to publish and use synthetic data](https://mlsanity.com/paper/2305.09235/) (2023)
- [ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Models](https://mlsanity.com/paper/2405.17724/) (2024)
- [Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare](https://mlsanity.com/paper/2401.13716/) (2024)
- [TabularARGN: A Flexible and Efficient Auto-Regressive Framework for Generating High-Fidelity Synthetic Data](https://mlsanity.com/paper/2501.12012/) (2025)
- [A Review on Generative AI Models for Synthetic Medical Text, Time Series, and Longitudinal Data](https://mlsanity.com/paper/2411.12274/) (2024)
- [Joint Relational Database Generation via Graph-Conditional Diffusion Models](https://mlsanity.com/paper/2505.16527/) (2025)
- [Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework](https://mlsanity.com/paper/2406.08311/) (2024)
- [LaTable: Towards Large Tabular Models](https://mlsanity.com/paper/2406.17673/) (2024)
- [FLAIM: AIM-based Synthetic Data Generation in the Federated Setting](https://mlsanity.com/paper/2310.03447/) (2023)
- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [Privacy Measurement in Tabular Synthetic Data: State of the Art and Future Research Directions](https://mlsanity.com/paper/2311.17453/) (2023)
- [RelDiff: Relational Data Generative Modeling with Graph-Based Diffusion Models](https://mlsanity.com/paper/2506.00710/) (2025)
- [The Minimum Information about CLinical Artificial Intelligence Checklist for Generative Modeling Research (MI-CLAIM-GEN)](https://mlsanity.com/paper/2403.02558/) (2024)
- [Contrastive Learning-Based privacy metrics in Tabular Synthetic Datasets](https://mlsanity.com/paper/2502.13833/) (2025)
- [Leveraging Vertical Public-Private Split for Improved Synthetic Data Generation](https://mlsanity.com/paper/2504.10987/) (2025)
- [MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular data](https://mlsanity.com/paper/2603.19185/) (2026)
- [On the Challenges of Deploying Privacy-Preserving Synthetic Data in the Enterprise](https://mlsanity.com/paper/2307.04208/) (2023)
- [Beyond a Single Mode: GAN Ensembles for Diverse Medical Data Generation](https://mlsanity.com/paper/2503.24258/) (2025)
- [STEB: In Search of the Best Evaluation Approach for Synthetic Time Series](https://mlsanity.com/paper/2505.21160/) (2025)
- [Efficacy of Image Similarity as a Metric for Augmenting Small Dataset Retinal Image Segmentation](https://mlsanity.com/paper/2507.04862/) (2025)
- [Democratizing Tabular Data Access with an Openunicodex2013Source Syntheticunicodex2013Data SDK](https://mlsanity.com/paper/2508.00718/) (2025)
- [A primer on synthetic health data](https://mlsanity.com/paper/2401.17653/) (2024)
- [Privacy-Preserving Tabular Synthetic Data Generation Using TabularARGN](https://mlsanity.com/paper/2508.06647/) (2025)
- [Using Synthetic Data to estimate the True Error is theoretically and practically doable](https://mlsanity.com/paper/2511.00964/) (2025)
- [Should I use Synthetic Data for That? An Analysis of the Suitability of Synthetic Data for Data Sharing and Augmentation](https://mlsanity.com/paper/2602.03791/) (2026)
- [CDMD: A Cross-Dataset Mixed-Type Diffusion Model for Tabular Data](https://mlsanity.com/paper/2609.39124/) (2026)

## Abstract (arXiv)

Generating synthetic data through generative models is gaining interest in the ML community and beyond. In the past, synthetic data was often regarded as a means to private data release, but a surge of recent papers explore how its potential reaches much further than this -- from creating more fair data to data augmentation, and from simulation to text generated by ChatGPT. In this perspective we explore whether, and how, synthetic data may become a dominant force in the machine learning world, promising a future where datasets can be tailored to individual needs. Just as importantly, we discuss which fundamental challenges the community needs to overcome for wider relevance and application of synthetic data -- the most important of which is quantifying how much we can trust any finding or prediction drawn from synthetic data.