# Sparse Flows: Pruning Continuous-depth Models

arXiv: https://arxiv.org/abs/2106.12718 · HTML: https://mlsanity.com/paper/2106.12718/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 79 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 53)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Residual Flows for Invertible Generative Modeling](https://mlsanity.com/paper/1906.02735/) (2019)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning](https://mlsanity.com/paper/1705.07761/) (2017)
- [Liquid Time-constant Networks](https://mlsanity.com/paper/2006.04439/) (2020)
- [Neural Importance Sampling](https://mlsanity.com/paper/1808.03856/) (2018)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)

## Built on by (most cited first, 13 of 13)

- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [On the Forward Invariance of Neural ODEs](https://mlsanity.com/paper/2210.04763/) (2022)
- [Sparsity in Continuous-Depth Neural Networks](https://mlsanity.com/paper/2210.14672/) (2022)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [Layer Collapse Can be Induced by Unstructured Pruning](https://mlsanity.com/paper/2404.16890/) (2024)
- [End-to-End Sensitivity-Based Filter Pruning](https://mlsanity.com/paper/2204.07412/) (2022)
- [Interpreting Neural Policies with Disentangled Tree Representations](https://mlsanity.com/paper/2210.06650/) (2022)
- [Pruning neural network models for gene regulatory dynamics using data and domain knowledge](https://mlsanity.com/paper/2403.04805/) (2024)
- [Pruning by Active Attention Manipulation](https://mlsanity.com/paper/2210.11114/) (2022)
- [Model Recycling Framework for Multi-Source Data-Free Supervised Transfer Learning](https://mlsanity.com/paper/2508.02039/) (2025)
- [Convergence, design and training of continuous-time dropout as a random batch method](https://mlsanity.com/paper/2510.13134/) (2025)

## Abstract (arXiv)

Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, pruning finds efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. We hope our results will invigorate further research into the performance-size trade-offs of modern continuous-depth models.