# Densely connected normalizing flows

arXiv: https://arxiv.org/abs/2106.04627 · HTML: https://mlsanity.com/paper/2106.04627/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 80 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)

## Built on by (most cited first, 30 of 49)

- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Consistency Flow Matching: Defining Straight Flows with Velocity Consistency](https://mlsanity.com/paper/2407.02398/) (2024)
- [Hierarchical Transformers Are More Efficient Language Models](https://mlsanity.com/paper/2110.13711/) (2021)
- [Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation](https://mlsanity.com/paper/2106.05527/) (2021)
- [Generating High Fidelity Data from Low-density Regions using Diffusion Models](https://mlsanity.com/paper/2203.17260/) (2022)
- [LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation](https://mlsanity.com/paper/2310.10769/) (2023)
- [normflows: A PyTorch Package for Normalizing Flows](https://mlsanity.com/paper/2302.12014/) (2023)
- [Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding](https://mlsanity.com/paper/2106.12566/) (2021)
- [Resampling Base Distributions of Normalizing Flows](https://mlsanity.com/paper/2110.15828/) (2021)
- [Maximum Likelihood Training of Implicit Nonlinear Diffusion Models](https://mlsanity.com/paper/2205.13699/) (2022)
- [Outlier detection by ensembling uncertainty with negative objectness](https://mlsanity.com/paper/2402.15374/) (2024)
- [Efficient-VDVAE: Less is more](https://mlsanity.com/paper/2203.13751/) (2022)
- [Transformer-VQ: Linear-Time Transformers via Vector Quantization](https://mlsanity.com/paper/2309.16354/) (2023)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask](https://mlsanity.com/paper/2309.04399/) (2023)
- [Boomerang: Local sampling on image manifolds using diffusion models](https://mlsanity.com/paper/2210.12100/) (2022)
- [ButterflyFlow: Building Invertible Layers with Butterfly Matrices](https://mlsanity.com/paper/2209.13774/) (2022)
- [Jet: A Modern Transformer-Based Normalizing Flow](https://mlsanity.com/paper/2412.15129/) (2024)
- [ASPIRE: Iterative Amortized Posterior Inference for Bayesian Inverse Problems](https://mlsanity.com/paper/2405.05398/) (2024)
- [Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data](https://mlsanity.com/paper/2112.12833/) (2021)
- [Scalable Normalizing Flows Enable Boltzmann Generators for Macromolecules](https://mlsanity.com/paper/2401.04246/) (2024)
- [Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows](https://mlsanity.com/paper/2305.18910/) (2023)
- [Hybrid Open-set Segmentation with Synthetic Negative Data](https://mlsanity.com/paper/2301.08555/) (2023)
- [LU-Net: Invertible Neural Networks Based on Matrix Factorization](https://mlsanity.com/paper/2302.10524/) (2023)
- [Diffusion Models with Deterministic Normalizing Flow Priors](https://mlsanity.com/paper/2309.01274/) (2023)
- [Score Mismatching for Generative Modeling](https://mlsanity.com/paper/2309.11043/) (2023)
- [Invertible Monotone Operators for Normalizing Flows](https://mlsanity.com/paper/2210.08176/) (2022)
- [Bootstrap Your Flow](https://mlsanity.com/paper/2111.11510/) (2021)
- [ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models](https://mlsanity.com/paper/2311.14097/) (2023)

## Abstract (arXiv)

Normalizing flows are bijective mappings between inputs and latent representations with a fully factorized distribution. They are very attractive due to exact likelihood valuation and efficient sampling. However, their effective capacity is often insufficient since the bijectivity constraint limits the model width. We address this issue by incrementally padding intermediate representations with noise. We precondition the noise in accordance with previous invertible units, which we describe as cross-unit coupling. Our invertible glow-like modules increase the model expressivity by fusing a densely connected block with Nystrom self-attention. We refer to our architecture as DenseFlow since both cross-unit and intra-module couplings rely on dense connectivity. Experiments show significant improvements due to the proposed contributions and reveal state-of-the-art density estimation under moderate computing budgets.