# Learning and Generalization in Overparameterized Normalizing Flows

arXiv: https://arxiv.org/abs/2106.10535 · HTML: https://mlsanity.com/paper/2106.10535/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 19 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [A vector-contraction inequality for Rademacher complexities](https://mlsanity.com/paper/1605.00251/) (2016)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators](https://mlsanity.com/paper/2006.11469/) (2020)
- [Generalization in Deep Networks: The Role of Distance from Initialization](https://mlsanity.com/paper/1901.01672/) (2019)
- [Improving Variational Auto-Encoders using Householder Flow](https://mlsanity.com/paper/1611.09630/) (2016)
- [Representational aspects of depth and conditioning in normalizing flows](https://mlsanity.com/paper/2010.01155/) (2020)
- [Normalizing Flows: An Introduction and Review of Current Methods](https://mlsanity.com/paper/1908.09257/) (2019)
- [Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysis](https://mlsanity.com/paper/1911.11983/) (2019)

## Built on by (most cited first, 1 of 1)

- [Semi-supervised Learning of Pushforwards For Domain Translation & Adaptation](https://mlsanity.com/paper/2304.08673/) (2023)

## Abstract (arXiv)

In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize, when trained using stochastic gradient descent with a sufficiently small learning rate and suitable initialization. In contrast, the benefit of overparameterization in unsupervised learning is not well understood. Normalizing flows (NFs) constitute an important class of models in unsupervised learning for sampling and density estimation. In this paper, we theoretically and empirically analyze these models when the underlying neural network is a one-hidden-layer overparametrized network. Our main contributions are two-fold: (1) On the one hand, we provide theoretical and empirical evidence that for constrained NFs (this class of NFs underlies many NF constructions) with the one-hidden-layer network, overparametrization hurts training. (2) On the other hand, we prove that unconstrained NFs, a recently introduced model, can efficiently learn any reasonable data distribution under minimal assumptions when the underlying network is overparametrized and has one hidden-layer.