# Rectangular Flows for Manifold Learning

arXiv: https://arxiv.org/abs/2106.01413 · HTML: https://mlsanity.com/paper/2106.01413/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 70 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Riemannian Continuous Normalizing Flows](https://mlsanity.com/paper/2006.10605/) (2020)
- [Flows for simultaneous manifold learning and density estimation](https://mlsanity.com/paper/2003.13913/) (2020)
- [Normalizing Flows on Tori and Spheres](https://mlsanity.com/paper/2002.02428/) (2020)
- [Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization](https://mlsanity.com/paper/2012.05942/) (2020)
- [Understanding and Mitigating Exploding Inverses in Invertible Neural Networks](https://mlsanity.com/paper/2006.09347/) (2020)
- [Normalizing Flows on Riemannian Manifolds](https://mlsanity.com/paper/1611.02304/) (2016)
- [Uncertainty in the Variational Information Bottleneck](https://mlsanity.com/paper/1807.00906/) (2018)
- [Implicit Normalizing Flows](https://mlsanity.com/paper/2103.09527/) (2021)
- [Regularized Autoencoders via Relaxed Injective Probability Flow](https://mlsanity.com/paper/2002.08927/) (2020)

## Built on by (most cited first, 30 of 50)

- [Riemannian Score-Based Generative Modelling](https://mlsanity.com/paper/2202.02763/) (2022)
- [Verifying the Union of Manifolds Hypothesis for Image Data](https://mlsanity.com/paper/2207.02862/) (2022)
- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows](https://mlsanity.com/paper/2106.05275/) (2021)
- [Explicitly Minimizing the Blur Error of Variational Autoencoders](https://mlsanity.com/paper/2304.05939/) (2023)
- [LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood](https://mlsanity.com/paper/2206.14882/) (2022)
- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds](https://mlsanity.com/paper/2211.15380/) (2022)
- [Free-form Flows: Make Any Architecture a Normalizing Flow](https://mlsanity.com/paper/2310.16624/) (2023)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [Lifting Architectural Constraints of Injective Flows](https://mlsanity.com/paper/2306.01843/) (2023)
- [Entropic Issues in Likelihood-Based OOD Detection](https://mlsanity.com/paper/2109.10794/) (2021)
- [Canonical normalizing flows for manifold learning](https://mlsanity.com/paper/2310.12743/) (2023)
- [A Review of Change of Variable Formulas for Generative Modeling](https://mlsanity.com/paper/2308.02652/) (2023)
- [Gradual Domain Adaptation via Normalizing Flows](https://mlsanity.com/paper/2206.11492/) (2022)
- [VQ-Flows: Vector Quantized Local Normalizing Flows](https://mlsanity.com/paper/2203.11556/) (2022)
- [Semi-Autoregressive Energy Flows: Exploring Likelihood-Free Training of Normalizing Flows](https://mlsanity.com/paper/2206.06672/) (2022)
- [Nonlinear Isometric Manifold Learning for Injective Normalizing Flows](https://mlsanity.com/paper/2203.03934/) (2022)
- [Denoising Deep Generative Models](https://mlsanity.com/paper/2212.01265/) (2022)
- [Diffusion map particle systems for generative modeling](https://mlsanity.com/paper/2304.00200/) (2023)
- [Flowification: Everything is a Normalizing Flow](https://mlsanity.com/paper/2205.15209/) (2022)
- [Neural Implicit Manifold Learning for Topology-Aware Density Estimation](https://mlsanity.com/paper/2206.11267/) (2022)
- [Implicit Variational Inference for High-Dimensional Posteriors](https://mlsanity.com/paper/2310.06643/) (2023)
- [Spread Flows for Manifold Modelling](https://mlsanity.com/paper/2109.14216/) (2021)
- [Principal Manifold Flows](https://mlsanity.com/paper/2202.07037/) (2022)
- [Deterministic training of generative autoencoders using invertible layers](https://mlsanity.com/paper/2205.09546/) (2022)
- [Particle-Based Score Estimation for State Space Model Learning in Autonomous Driving](https://mlsanity.com/paper/2212.06968/) (2022)
- [Generative Invertible Quantum Neural Networks](https://mlsanity.com/paper/2302.12906/) (2023)
- [Learning Geometry and Topology via Multi-Chart Flows](https://mlsanity.com/paper/2505.24665/) (2025)

## Abstract (arXiv)

Normalizing flows are invertible neural networks with tractable change-of-volume terms, which allow optimization of their parameters to be efficiently performed via maximum likelihood. However, data of interest are typically assumed to live in some (often unknown) low-dimensional manifold embedded in a high-dimensional ambient space. The result is a modelling mismatch since -- by construction -- the invertibility requirement implies high-dimensional support of the learned distribution. Injective flows, mappings from low- to high-dimensional spaces, aim to fix this discrepancy by learning distributions on manifolds, but the resulting volume-change term becomes more challenging to evaluate. Current approaches either avoid computing this term entirely using various heuristics, or assume the manifold is known beforehand and therefore are not widely applicable. Instead, we propose two methods to tractably calculate the gradient of this term with respect to the parameters of the model, relying on careful use of automatic differentiation and techniques from numerical linear algebra. Both approaches perform end-to-end nonlinear manifold learning and density estimation for data projected onto this manifold. We study the trade-offs between our proposed methods, empirically verify that we outperform approaches ignoring the volume-change term by more accurately learning manifolds and the corresponding distributions on them, and show promising results on out-of-distribution detection. Our code is available at https://github.com/layer6ai-labs/rectangular-flows.