# Residual Flows for Invertible Generative Modeling

arXiv: https://arxiv.org/abs/1906.02735 · HTML: https://mlsanity.com/paper/1906.02735/

## Key facts

- Cited by 257 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 30 of 257)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models](https://mlsanity.com/paper/2305.18455/) (2023)
- [Score-Based Generative Modeling with Critically-Damped Langevin Diffusion](https://mlsanity.com/paper/2112.07068/) (2021)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Path Integral Sampler: a stochastic control approach for sampling](https://mlsanity.com/paper/2111.15141/) (2021)
- [Consistency Flow Matching: Defining Straight Flows with Velocity Consistency](https://mlsanity.com/paper/2407.02398/) (2024)
- [Independent mechanism analysis, a new concept?](https://mlsanity.com/paper/2106.05200/) (2021)
- [Poisson Flow Generative Models](https://mlsanity.com/paper/2209.11178/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency](https://mlsanity.com/paper/2510.08431/) (2025)
- [Diffusion Normalizing Flow](https://mlsanity.com/paper/2110.07579/) (2021)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Efficient and Accurate Gradients for Neural SDEs](https://mlsanity.com/paper/2105.13493/) (2021)
- [One-Step Diffusion Distillation through Score Implicit Matching](https://mlsanity.com/paper/2410.16794/) (2024)
- [Neural Conservation Laws: A Divergence-Free Perspective](https://mlsanity.com/paper/2210.01741/) (2022)
- [Moser Flow: Divergence-based Generative Modeling on Manifolds](https://mlsanity.com/paper/2108.08052/) (2021)
- [Verifying the Union of Manifolds Hypothesis for Image Data](https://mlsanity.com/paper/2207.02862/) (2022)
- [Towards Blind Watermarking: Combining Invertible and Non-invertible Mechanisms](https://mlsanity.com/paper/2212.12678/) (2022)
- [LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation](https://mlsanity.com/paper/2310.10769/) (2023)
- [360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model](https://mlsanity.com/paper/2401.06578/) (2024)
- [normflows: A PyTorch Package for Normalizing Flows](https://mlsanity.com/paper/2302.12014/) (2023)
- [Optimizing Functionals on the Space of Probabilities with Input Convex Neural Networks](https://mlsanity.com/paper/2106.00774/) (2021)
- [Rectangular Flows for Manifold Learning](https://mlsanity.com/paper/2106.01413/) (2021)
- [Densely connected normalizing flows](https://mlsanity.com/paper/2106.04627/) (2021)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)
- [How Smooth Is Attention?](https://mlsanity.com/paper/2312.14820/) (2023)

## Abstract (arXiv)

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforcing invertibility. However, prior work trained invertible residual networks for density estimation by relying on biased log-density estimates whose bias increased with the network's expressiveness. We give a tractable unbiased estimate of the log density using a "Russian roulette" estimator, and reduce the memory required during training by using an alternative infinite series for the gradient. Furthermore, we improve invertible residual blocks by proposing the use of activation functions that avoid derivative saturation and generalizing the Lipschitz condition to induced mixed norms. The resulting approach, called Residual Flows, achieves state-of-the-art performance on density estimation amongst flow-based models, and outperforms networks that use coupling blocks at joint generative and discriminative modeling.