# Representational aspects of depth and conditioning in normalizing flows

arXiv: https://arxiv.org/abs/2010.01155 · HTML: https://mlsanity.com/paper/2010.01155/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 25 of 25)

- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [Free-form Flows: Make Any Architecture a Normalizing Flow](https://mlsanity.com/paper/2310.16624/) (2023)
- [On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows](https://mlsanity.com/paper/2402.06578/) (2024)
- [On the expressivity of bi-Lipschitz normalizing flows](https://mlsanity.com/paper/2107.07232/) (2021)
- [Multiscale Flow for Robust and Optimal Cosmological Analysis](https://mlsanity.com/paper/2306.04689/) (2023)
- [Jacobian Determinant of Normalizing Flows](https://mlsanity.com/paper/2102.06539/) (2021)
- [Universal Approximation of Residual Flows in Maximum Mean Discrepancy](https://mlsanity.com/paper/2103.05793/) (2021)
- [Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows](https://mlsanity.com/paper/2107.02951/) (2021)
- [Enhancing Accuracy in Generative Models via Knowledge Transfer](https://mlsanity.com/paper/2405.16837/) (2024)
- [Heavy-tailed Sampling via Transformed Unadjusted Langevin Algorithm](https://mlsanity.com/paper/2201.08349/) (2022)
- [Whitening Convergence Rate of Coupling-based Normalizing Flows](https://mlsanity.com/paper/2210.14032/) (2022)
- [MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction](https://mlsanity.com/paper/2402.12238/) (2024)
- [Stable Training of Normalizing Flows for High-dimensional Variational Inference](https://mlsanity.com/paper/2402.16408/) (2024)
- [On the Convergence Rate of Gaussianization with Random Rotations](https://mlsanity.com/paper/2306.13520/) (2023)
- [A Unified Generative-Predictive Framework for Deterministic Inverse Design](https://mlsanity.com/paper/2512.15746/) (2025)
- [Learning to Generate All Feasible Actions](https://mlsanity.com/paper/2301.11461/) (2023)
- [Learning and Generalization in Overparameterized Normalizing Flows](https://mlsanity.com/paper/2106.10535/) (2021)
- [Perturbation-Assisted Sample Synthesis: A Novel Approach for Uncertainty Quantification](https://mlsanity.com/paper/2305.18671/) (2023)
- [VeriFlow: Modeling Distributions for Neural Network Verification](https://mlsanity.com/paper/2406.14265/) (2024)
- [Asymptotically exact variational flows via involutive MCMC kernels](https://mlsanity.com/paper/2506.02162/) (2025)
- [Analytic Bijections for Smooth and Interpretable Normalizing Flows](https://mlsanity.com/paper/2601.10774/) (2026)
- [Estimation of Counterfactual Interventions under Uncertainties](https://mlsanity.com/paper/2309.08332/) (2023)
- [A Novel Framework Using Variational Inference with Normalizing Flows to Train Transport Reversible Jump Proposals](https://mlsanity.com/paper/2512.12742/) (2025)
- [Priors learned from legacy reconstructions inherit undetectable overconfidence](https://mlsanity.com/paper/2607.21721/) (2026)

## Abstract (arXiv)

Normalizing flows are among the most popular paradigms in generative modeling, especially for images, primarily because we can efficiently evaluate the likelihood of a data point. This is desirable both for evaluating the fit of a model, and for ease of training, as maximizing the likelihood can be done by gradient descent. However, training normalizing flows comes with difficulties as well: models which produce good samples typically need to be extremely deep -- which comes with accompanying vanishing/exploding gradient problems. A very related problem is that they are often poorly conditioned: since they are parametrized as invertible maps from ℝd → ℝd, and typical training data like images intuitively is lower-dimensional, the learned maps often have Jacobians that are close to being singular. In our paper, we tackle representational aspects around depth and conditioning of normalizing flows: both for general invertible architectures, and for a particular common architecture, affine couplings. We prove that Theta(1) affine coupling layers suffice to exactly represent a permutation or 1 × 1 convolution, as used in GLOW, showing that representationally the choice of partition is not a bottleneck for depth. We also show that shallow affine coupling networks are universal approximators in Wasserstein distance if ill-conditioning is allowed, and experimentally investigate related phenomena involving padding. Finally, we show a depth lower bound for general flow architectures with few neurons per layer and bounded Lipschitz constant.