# Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution

arXiv: https://arxiv.org/abs/2307.16422 · HTML: https://mlsanity.com/paper/2307.16422/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 68 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 30 of 42)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling](https://mlsanity.com/paper/2106.01357/) (2021)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [Riemannian Score-Based Generative Modelling](https://mlsanity.com/paper/2202.02763/) (2022)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning](https://mlsanity.com/paper/1705.07761/) (2017)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Generative Modeling with Denoising Auto-Encoders and Langevin Sampling](https://mlsanity.com/paper/2002.00107/) (2020)
- [Stabilizing Training of Generative Adversarial Networks through Regularization](https://mlsanity.com/paper/1705.09367/) (2017)
- [Measuring Forgetting of Memorized Training Examples](https://mlsanity.com/paper/2207.00099/) (2022)
- [Ambient Diffusion: Learning Clean Distributions from Corrupted Data](https://mlsanity.com/paper/2305.19256/) (2023)
- [Quant GANs: Deep Generation of Financial Time Series](https://mlsanity.com/paper/1907.06673/) (2019)
- [Exactly Computing the Local Lipschitz Constant of ReLU Networks](https://mlsanity.com/paper/2003.01219/) (2020)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks](https://mlsanity.com/paper/1712.10321/) (2017)
- [How Well Generative Adversarial Networks Learn Distributions](https://mlsanity.com/paper/1811.03179/) (2018)
- [Mol-CycleGAN - a generative model for molecular optimization](https://mlsanity.com/paper/1902.02119/) (2019)
- [On the regularization of Wasserstein GANs](https://mlsanity.com/paper/1709.08894/) (2017)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model](https://mlsanity.com/paper/1909.04711/) (2019)
- [An error analysis of generative adversarial networks for learning distributions](https://mlsanity.com/paper/2105.13010/) (2021)
- [Indeterminacy in Generative Models: Characterization and Strong Identifiability](https://mlsanity.com/paper/2206.00801/) (2022)
- [Towards GAN Benchmarks Which Require Generalization](https://mlsanity.com/paper/2001.03653/) (2020)

## Built on by (most cited first, 3 of 3)

- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [A Good Score Does not Lead to A Good Generative Model](https://mlsanity.com/paper/2401.04856/) (2024)
- [On the minimax optimality of Flow Matching through the connection to kernel density estimation](https://mlsanity.com/paper/2504.13336/) (2025)

## Abstract (arXiv)

This paper explores the problem of generative modeling, aiming to simulate diverse examples from an unknown distribution based on observed examples. While recent studies have focused on quantifying the statistical precision of popular algorithms, there is a lack of mathematical evaluation regarding the non-replication of observed examples and the creativity of the generative model. We present theoretical insights into this aspect, demonstrating that the Wasserstein GAN, constrained to left-invertible push-forward maps, generates distributions that avoid replication and significantly deviate from the empirical distribution. Importantly, we show that left-invertibility achieves this without compromising the statistical optimality of the resulting generator. Our most important contribution provides a finite-sample lower bound on the Wasserstein-1 distance between the generative distribution and the empirical one. We also establish a finite-sample upper bound on the distance between the generative distribution and the true data-generating one. Both bounds are explicit and show the impact of key parameters such as sample size, dimensions of the ambient and latent spaces, noise level, and smoothness measured by the Lipschitz constant.