# Invertible DenseNets with Concatenated LipSwish

arXiv: https://arxiv.org/abs/2102.02694 · HTML: https://mlsanity.com/paper/2102.02694/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 4 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://mlsanity.com/paper/1603.05201/) (2016)
- [Sylvester Normalizing Flows for Variational Inference](https://mlsanity.com/paper/1803.05649/) (2018)
- [Excessive Invariance Causes Adversarial Vulnerability](https://mlsanity.com/paper/1811.00401/) (2018)
- [Block Neural Autoregressive Flow](https://mlsanity.com/paper/1904.04676/) (2019)
- [Improving Variational Auto-Encoders using Householder Flow](https://mlsanity.com/paper/1611.09630/) (2016)
- [Hybrid Models with Deep and Invertible Features](https://mlsanity.com/paper/1902.02767/) (2019)
- [Understanding the Limitations of Conditional Generative Models](https://mlsanity.com/paper/1906.01171/) (2019)
- [Learning Discrete Distributions by Dequantization](https://mlsanity.com/paper/2001.11235/) (2020)
- [The Convolution Exponential and Generalized Sylvester Flows](https://mlsanity.com/paper/2006.01910/) (2020)
- [High-Dimensional Probability Estimation with Deep Density Models](https://mlsanity.com/paper/1302.5125/) (2013)

## Built on by (most cited first, 17 of 17)

- [Densely connected normalizing flows](https://mlsanity.com/paper/2106.04627/) (2021)
- [Learning Data Representations with Joint Diffusion Models](https://mlsanity.com/paper/2301.13622/) (2023)
- [Estimating Treatment Effects from Irregular Time Series Observations with Hidden Confounders](https://mlsanity.com/paper/2303.02320/) (2023)
- [ButterflyFlow: Building Invertible Layers with Butterfly Matrices](https://mlsanity.com/paper/2209.13774/) (2022)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Invertible Monotone Operators for Normalizing Flows](https://mlsanity.com/paper/2210.08176/) (2022)
- [Lagrangian Flow Networks for Conservation Laws](https://mlsanity.com/paper/2305.16846/) (2023)
- [Entropy-Informed Weighting Channel Normalizing Flow for Deep Generative Models](https://mlsanity.com/paper/2407.04958/) (2024)
- [Computing excited states of molecules using normalizing flows](https://mlsanity.com/paper/2308.16468/) (2023)
- [Estimating Treatment Effects in Continuous Time with Hidden Confounders](https://mlsanity.com/paper/2302.09446/) (2023)
- [LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers](https://mlsanity.com/paper/2410.22258/) (2024)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Discouraging posterior collapse in hierarchical Variational Autoencoders using context](https://mlsanity.com/paper/2302.09976/) (2023)
- [Injective flows for star-like manifolds](https://mlsanity.com/paper/2406.09116/) (2024)
- [CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices](https://mlsanity.com/paper/2510.25323/) (2025)
- [Attentive Contractive Flow with Lipschitz-constrained Self-Attention](https://mlsanity.com/paper/2109.12135/) (2021)
- [Posterior shape models revisited: Improving 3D reconstructions from partial data using target specific models](https://mlsanity.com/paper/2509.05776/) (2025)

## Abstract (arXiv)

We introduce Invertible Dense Networks (i-DenseNets), a more parameter efficient extension of Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce invertibility of the network by satisfying the Lipschitz constant. Furthermore, we propose a learnable weighted concatenation, which not only improves the model performance but also indicates the importance of the concatenated weighted representation. Additionally, we introduce the Concatenated LipSwish as activation function, for which we show how to enforce the Lipschitz condition and which boosts performance. The new architecture, i-DenseNet, out-performs Residual Flow and other flow-based models on density estimation evaluated in bits per dimension, where we utilize an equal parameter budget. Moreover, we show that the proposed model out-performs Residual Flows when trained as a hybrid model where the model is both a generative and a discriminative model.