Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution
Not on the map. Why?
Abstract (arXiv)
This paper explores the problem of generative modeling, aiming to simulate diverse examples from an unknown distribution based on observed examples. While recent studies have focused on quantifying the statistical precision of popular algorithms, there is a lack of mathematical evaluation regarding the non-replication of observed examples and the creativity of the generative model. We present theoretical insights into this aspect, demonstrating that the Wasserstein GAN, constrained to left-invertible push-forward maps, generates distributions that avoid replication and significantly deviate from the empirical distribution. Importantly, we show that left-invertibility achieves this without compromising the statistical optimality of the resulting generator. Our most important contribution provides a finite-sample lower bound on the Wasserstein-1 distance between the generative distribution and the empirical one. We also establish a finite-sample upper bound on the distance between the generative distribution and the true data-generating one. Both bounds are explicit and show the impact of key parameters such as sample size, dimensions of the ambient and latent spaces, noise level, and smoothness measured by the Lipschitz constant.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 68 references that are in the map. First posted on arXiv 31 Jul 2023.
Builds on
42 of 68 references.
References · 42
- 2022Generative Adversarial Networkscited by 5,603
- 2017Improved Training of Wasserstein GANscited by 2,223
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2023Extracting Training Data from Diffusion Modelscited by 723
- 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
- 2017Learning Generative Models with Sinkhorn Divergencescited by 310
- 2016Adversarially Learned Inferencecited by 216
- 2022Riemannian Score-Based Generative Modellingcited by 216
- 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
- 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
30 more
- 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
- 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
- 2017Stabilizing Training of Generative Adversarial Networks through Regularizationcited by 118
- 2020Generative Modeling with Denoising Auto-Encoders and Langevin Samplingcited by 118
- 2022Measuring Forgetting of Memorized Training Examplescited by 118
- 2023Ambient Diffusion: Learning Clean Distributions from Corrupted Datacited by 108
- 2019Quant GANs: Deep Generation of Financial Time Seriescited by 107
- 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
- 2016Medical Image Synthesis with Context-Aware Generative Adversarial Networkscited by 91
- 2017CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networkscited by 78
- 2018How Well Generative Adversarial Networks Learn Distributionscited by 78
- 2019Mol-CycleGAN - a generative model for molecular optimizationcited by 66
- 2017On the regularization of Wasserstein GANscited by 65
- 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
- 2019Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Modelcited by 45
- 2021An error analysis of generative adversarial networks for learning distributionscited by 35
- 2022Indeterminacy in Generative Models: Characterization and Strong Identifiabilitycited by 33
- 2020Towards GAN Benchmarks Which Require Generalizationcited by 31
- 2018Approximability of Discriminators Implies Diversity in GANscited by 28
- 2020Some Theoretical Insights into Wasserstein GANscited by 28
- 2018Some Theoretical Properties of GANscited by 27
- 2020Statistical guarantees for generative models without dominationcited by 27
- 2018BourGAN: Generative Networks with Metric Embeddingscited by 22
- 2019Nonparametric Density Estimation & Convergence Rates for GANs under Besov IPM Lossescited by 22
- 2021On the capacity of deep generative networks for approximating distributionscited by 19
- 2020Generalization Properties of Optimal Transport GANs with Latent Distribution Learningcited by 13
- 2021Rates of convergence for density estimation with generative adversarial networkscited by 13
- 2021A likelihood approach to nonparametric estimation of a singular distribution using deep generative modelscited by 12
- 2021Provable Lipschitz Certification for Generative Modelscited by 7
- 2022Minimax Optimality (Probably) Doesn't Imply Distribution Learning for GANscited by 6
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.