# Understanding and Mitigating Exploding Inverses in Invertible Neural Networks

arXiv: https://arxiv.org/abs/2006.09347 · HTML: https://mlsanity.com/paper/2006.09347/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 30 of 71)

- [Rectangular Flows for Manifold Learning](https://mlsanity.com/paper/2106.01413/) (2021)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections](https://mlsanity.com/paper/2404.02954/) (2024)
- [Resampling Base Distributions of Normalizing Flows](https://mlsanity.com/paper/2110.15828/) (2021)
- [Can Push-forward Generative Models Fit Multimodal Distributions?](https://mlsanity.com/paper/2206.14476/) (2022)
- [Stochastic Normalizing Flows for Inverse Problems: a Markov Chains Viewpoint](https://mlsanity.com/paper/2109.11375/) (2021)
- [Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows](https://mlsanity.com/paper/2106.05275/) (2021)
- [Diagnosing and Fixing Manifold Overfitting in Deep Generative Models](https://mlsanity.com/paper/2204.07172/) (2022)
- [Universality laws for Gaussian mixtures in generalized linear models](https://mlsanity.com/paper/2302.08933/) (2023)
- [iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression](https://mlsanity.com/paper/2103.16211/) (2021)
- [iFlow: Numerically Invertible Flows for Efficient Lossless Compression via a Uniform Coder](https://mlsanity.com/paper/2111.00965/) (2021)
- [Generalized Normalizing Flows via Markov Chains](https://mlsanity.com/paper/2111.12506/) (2021)
- [Regularising Inverse Problems with Generative Machine Learning Models](https://mlsanity.com/paper/2107.11191/) (2021)
- [A Geometric Explanation of the Likelihood OOD Detection Paradox](https://mlsanity.com/paper/2403.18910/) (2024)
- [On the expressivity of bi-Lipschitz normalizing flows](https://mlsanity.com/paper/2107.07232/) (2021)
- [Flow-based Self-supervised Density Estimation for Anomalous Sound Detection](https://mlsanity.com/paper/2103.08801/) (2021)
- [Cycle-free CycleGAN using Invertible Generator for Unsupervised Low-Dose CT Denoising](https://mlsanity.com/paper/2104.08538/) (2021)
- [Conditional Generative Models are Provably Robust: Pointwise Guarantees for Bayesian Inverse Problems](https://mlsanity.com/paper/2303.15845/) (2023)
- [TrackFlow: Multi-Object Tracking with Normalizing Flows](https://mlsanity.com/paper/2308.11513/) (2023)
- [Marginal Tail-Adaptive Normalizing Flows](https://mlsanity.com/paper/2206.10311/) (2022)
- [Out-of-Distribution Detection with Reconstruction Error and Typicality-based Penalty](https://mlsanity.com/paper/2212.12641/) (2022)
- [Principal Component Density Estimation for Scenario Generation Using Normalizing Flows](https://mlsanity.com/paper/2104.10410/) (2021)
- [VQ-Flows: Vector Quantized Local Normalizing Flows](https://mlsanity.com/paper/2203.11556/) (2022)
- [Insights from Generative Modeling for Neural Video Compression](https://mlsanity.com/paper/2107.13136/) (2021)
- [Learning from small data sets: Patch-based regularizers in inverse problems for image reconstruction](https://mlsanity.com/paper/2312.16611/) (2023)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks](https://mlsanity.com/paper/2107.06304/) (2021)
- [On the Robustness of Normalizing Flows for Inverse Problems in Imaging](https://mlsanity.com/paper/2212.04319/) (2022)
- [Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows](https://mlsanity.com/paper/2107.02951/) (2021)
- [Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning](https://mlsanity.com/paper/2311.06481/) (2023)

## Abstract (arXiv)

Invertible neural networks (INNs) have been used to design generative models, implement memory-saving gradient computation, and solve inverse problems. In this work, we show that commonly-used INN architectures suffer from exploding inverses and are thus prone to becoming numerically non-invertible. Across a wide range of INN use-cases, we reveal failures including the non-applicability of the change-of-variables formula on in- and out-of-distribution (OOD) data, incorrect gradients for memory-saving backprop, and the inability to sample from normalizing flow models. We further derive bi-Lipschitz properties of atomic building blocks of common architectures. These insights into the stability of INNs then provide ways forward to remedy these failures. For tasks where local invertibility is sufficient, like memory-saving backprop, we propose a flexible and efficient regularizer. For problems where global invertibility is necessary, such as applying normalizing flows on OOD data, we show the importance of designing stable INN building blocks.