# Whitening Convergence Rate of Coupling-based Normalizing Flows

arXiv: https://arxiv.org/abs/2210.14032 · HTML: https://mlsanity.com/paper/2210.14032/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Neural Importance Sampling](https://mlsanity.com/paper/1808.03856/) (2018)
- [Unconstrained Monotonic Neural Networks](https://mlsanity.com/paper/1908.05164/) (2019)
- [Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators](https://mlsanity.com/paper/2006.11469/) (2020)
- [Latent Normalizing Flows for Discrete Sequences](https://mlsanity.com/paper/1901.10548/) (2019)
- [Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models](https://mlsanity.com/paper/2002.07101/) (2020)
- [Sum-of-Squares Polynomial Flow](https://mlsanity.com/paper/1905.02325/) (2019)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [Cubic-Spline Flows](https://mlsanity.com/paper/1906.02145/) (2019)
- [Universal Approximation Property of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2012.02414/) (2020)
- [Universal approximation property of invertible neural networks](https://mlsanity.com/paper/2204.07415/) (2022)
- [Representational aspects of depth and conditioning in normalizing flows](https://mlsanity.com/paper/2010.01155/) (2020)
- [Gaussianization Flows](https://mlsanity.com/paper/2003.01941/) (2020)
- [Learning Robust Models Using The Principle of Independent Causal Mechanisms](https://mlsanity.com/paper/2010.07167/) (2020)
- [Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows](https://mlsanity.com/paper/2107.02951/) (2021)
- [Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks](https://mlsanity.com/paper/1903.03441/) (2019)

## Built on by (most cited first, 6 of 6)

- [JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models](https://mlsanity.com/paper/2302.09125/) (2023)
- [Free-form Flows: Make Any Architecture a Normalizing Flow](https://mlsanity.com/paper/2310.16624/) (2023)
- [Lifting Architectural Constraints of Injective Flows](https://mlsanity.com/paper/2306.01843/) (2023)
- [On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows](https://mlsanity.com/paper/2402.06578/) (2024)
- [On the Convergence Rate of Gaussianization with Random Rotations](https://mlsanity.com/paper/2306.13520/) (2023)
- [On the Approximation of Bi-Lipschitz Maps by Invertible Neural Networks](https://mlsanity.com/paper/2308.09367/) (2023)

## Abstract (arXiv)

Coupling-based normalizing flows (e.g. RealNVP) are a popular family of normalizing flow architectures that work surprisingly well in practice. This calls for theoretical understanding. Existing work shows that such flows weakly converge to arbitrary data distributions. However, they make no statement about the stricter convergence criterion used in practice, the maximum likelihood loss. For the first time, we make a quantitative statement about this kind of convergence: We prove that all coupling-based normalizing flows perform whitening of the data distribution (i.e. diagonalize the covariance matrix) and derive corresponding convergence bounds that show a linear convergence rate in the depth of the flow. Numerical experiments demonstrate the implications of our theory and point at open questions.