# A likelihood approach to nonparametric estimation of a singular distribution using deep generative models

arXiv: https://arxiv.org/abs/2105.04046 · HTML: https://mlsanity.com/paper/2105.04046/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 104 references that are in the map.
- First posted on arXiv 9 May 2021.

## Builds on (most cited first, 30 of 41)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [How Well Generative Adversarial Networks Learn Distributions](https://mlsanity.com/paper/1811.03179/) (2018)
- [Deep ReLU network approximation of functions on a manifold](https://mlsanity.com/paper/1908.00695/) (2019)
- [Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recovery](https://mlsanity.com/paper/1908.01842/) (2019)
- [Deep Neural Networks Learn Non-Smooth Functions Effectively](https://mlsanity.com/paper/1802.04474/) (2018)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [On the Discrimination-Generalization Tradeoff in GANs](https://mlsanity.com/paper/1711.02771/) (2017)
- [Reinterpreting Importance-Weighted Autoencoders](https://mlsanity.com/paper/1704.02916/) (2017)
- [Smooth function approximation by deep neural networks with general activation functions](https://mlsanity.com/paper/1906.06903/) (2019)
- [Approximation and Convergence Properties of Generative Adversarial Learning](https://mlsanity.com/paper/1705.08991/) (2017)
- [Approximability of Discriminators Implies Diversity in GANs](https://mlsanity.com/paper/1806.10586/) (2018)

## Built on by (most cited first, 12 of 12)

- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds](https://mlsanity.com/paper/2302.13183/) (2023)
- [Denoising Deep Generative Models](https://mlsanity.com/paper/2212.01265/) (2022)
- [A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models](https://mlsanity.com/paper/2410.02025/) (2024)
- [Deep Generative Models: Complexity, Dimensionality, and Approximation](https://mlsanity.com/paper/2504.00820/) (2025)
- [Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution](https://mlsanity.com/paper/2307.16422/) (2023)
- [Is an encoder within reach?](https://mlsanity.com/paper/2206.01552/) (2022)
- [Knowledge Distillation of Uncertainty using Deep Latent Factor Model](https://mlsanity.com/paper/2510.19290/) (2025)
- [Distribution estimation and change-point estimation for time series via DNN-based GANs](https://mlsanity.com/paper/2211.14577/) (2022)
- [Variational bagging: a robust approach for Bayesian uncertainty quantification](https://mlsanity.com/paper/2511.20594/) (2025)
- [Generative Model via Quantile Assignment](https://mlsanity.com/paper/2602.18216/) (2026)

## Abstract (arXiv)

We investigate statistical properties of a likelihood approach to nonparametric estimation of a singular distribution using deep generative models. More specifically, a deep generative model is used to model high-dimensional data that are assumed to concentrate around some low-dimensional structure. Estimating the distribution supported on this low-dimensional structure, such as a low-dimensional manifold, is challenging due to its singularity with respect to the Lebesgue measure in the ambient space. In the considered model, a usual likelihood approach can fail to estimate the target distribution consistently due to the singularity. We prove that a novel and effective solution exists by perturbing the data with an instance noise, which leads to consistent estimation of the underlying distribution with desirable convergence rates. We also characterize the class of distributions that can be efficiently estimated via deep generative models. This class is sufficiently general to contain various structured distributions such as product distributions, classically smooth distributions and distributions supported on a low-dimensional manifold. Our analysis provides some insights on how deep generative models can avoid the curse of dimensionality for nonparametric distribution estimation. We conduct a thorough simulation study and real data analysis to empirically demonstrate that the proposed data perturbation technique improves the estimation performance significantly.