# Learning Distributions Generated by One-Layer ReLU Networks

arXiv: https://arxiv.org/abs/1909.01812 · HTML: https://mlsanity.com/paper/1909.01812/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2019.

## Built on by (most cited first, 10 of 10)

- [Efficient Algorithms for Learning from Coarse Labels](https://mlsanity.com/paper/2108.09805/) (2021)
- [Agnostic Learning of General ReLU Activation Using Gradient Descent](https://mlsanity.com/paper/2208.02711/) (2022)
- [Learning (Very) Simple Generative Models Is Hard](https://mlsanity.com/paper/2205.16003/) (2022)
- [Generalization Error of GAN from the Discriminator's Perspective](https://mlsanity.com/paper/2107.03633/) (2021)
- [A Mathematical Framework for Learning Probability Distributions](https://mlsanity.com/paper/2212.11481/) (2022)
- [Lower Bounds on the Total Variation Distance Between Mixtures of Two Gaussians](https://mlsanity.com/paper/2109.01064/) (2021)
- [Sobolev acceleration for neural networks](https://mlsanity.com/paper/2509.19773/) (2025)
- [Learning High-dimensional Gaussians from Censored Data](https://mlsanity.com/paper/2504.19446/) (2025)
- [Copula-Based Density Estimation Models for Multivariate Zero-Inflated Continuous Data](https://mlsanity.com/paper/2304.00537/) (2023)
- [Mean Testing under Truncation beyond Gaussian](https://mlsanity.com/paper/2605.01335/) (2026)

## Abstract (arXiv)

We consider the problem of estimating the parameters of a d-dimensional rectified Gaussian distribution from i.i.d. samples. A rectified Gaussian distribution is defined by passing a standard Gaussian distribution through a one-layer ReLU neural network. We give a simple algorithm to estimate the parameters (i.e., the weight matrix and bias vector of the ReLU neural network) up to an error ε||W||F using tildeO(1/ε2) samples and tildeO(d2/ε2) time (log factors are ignored for simplicity). This implies that we can estimate the distribution up to ε in total variation distance using tildeO(kappa2d2/ε2) samples, where kappa is the condition number of the covariance matrix. Our only assumption is that the bias vector is non-negative. Without this non-negativity assumption, we show that estimating the bias vector within any error requires the number of samples at least exponential in the infinity norm of the bias vector. Our algorithm is based on the key observation that vector norms and pairwise angles can be estimated separately. We use a recent result on learning from truncated samples. We also prove two sample complexity lower bounds: Omega(1/ε2) samples are required to estimate the parameters up to error ε, while Omega(d/ε2) samples are necessary to estimate the distribution up to ε in total variation distance. The first lower bound implies that our algorithm is optimal for parameter estimation. Finally, we show an interesting connection between learning a two-layer generative model and non-negative matrix factorization. Experimental results are provided to support our analysis.