# An Operator Theoretic View on Pruning Deep Neural Networks

arXiv: https://arxiv.org/abs/2110.14856 · HTML: https://mlsanity.com/paper/2110.14856/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [A Signal Propagation Perspective for Pruning Neural Networks at Initialization](https://mlsanity.com/paper/1906.06307/) (2019)
- [Optimizing Neural Networks via Koopman Operator Theory](https://mlsanity.com/paper/2006.02361/) (2020)
- [Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning](https://mlsanity.com/paper/2007.08243/) (2020)
- [Applications of Koopman Mode Analysis to Neural Networks](https://mlsanity.com/paper/2006.11765/) (2020)
- [Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition](https://mlsanity.com/paper/2006.14371/) (2020)
- [Predicting the Critical Number of Layers for Hierarchical Support Vector Regression](https://mlsanity.com/paper/2012.11734/) (2020)

## Built on by (most cited first, 18 of 18)

- [Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs](https://mlsanity.com/paper/2310.02619/) (2023)
- [Multifactor Sequential Disentanglement via Structured Koopman Autoencoders](https://mlsanity.com/paper/2303.17264/) (2023)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Koopman-based generalization bound: New aspect for full-rank weights](https://mlsanity.com/paper/2302.05825/) (2023)
- [Identifying Equivalent Training Dynamics](https://mlsanity.com/paper/2302.09160/) (2023)
- [QuACK: Accelerating Gradient-Based Quantum Optimization with Koopman Operator Learning](https://mlsanity.com/paper/2211.01365/) (2022)
- [Enhancing Neural Training via a Correlated Dynamics Model](https://mlsanity.com/paper/2312.13247/) (2023)
- [Enhanced Pruning Strategy for Multi-Component Neural Architectures Using Component-Aware Graph Analysis](https://mlsanity.com/paper/2504.13296/) (2025)
- [Hierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory](https://mlsanity.com/paper/2510.12220/) (2025)
- [Koopman operator theory: fundamentals, control, and applications](https://mlsanity.com/paper/2607.01819/) (2026)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)
- [Ensemble Mask Networks](https://mlsanity.com/paper/2309.06382/) (2023)
- [Glocal Hypergradient Estimation with Koopman Operator](https://mlsanity.com/paper/2402.02741/) (2024)
- [Accelerating Inference of Networks in the Frequency Domain](https://mlsanity.com/paper/2410.04342/) (2024)
- [COMponent-Aware Pruning for Accelerated Control Tasks in Latent Space Models](https://mlsanity.com/paper/2508.08144/) (2025)
- [Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization](https://mlsanity.com/paper/2601.17987/) (2026)
- [Man, Machine, and Mathematics](https://mlsanity.com/paper/2604.27052/) (2026)
- [Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability](https://mlsanity.com/paper/2608.10172/) (2026)

## Abstract (arXiv)

The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the na"ive approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especially pre-convergence, and its relation to other pruning methods, such as gradient based pruning, are outstanding open questions in the field that are in need of being addressed. We make use of recent advances in dynamical systems theory, namely Koopman operator theory, to define a new class of theoretically motivated pruning algorithms. We show that these algorithms can be equivalent to magnitude and gradient based pruning, unifying these seemingly disparate methods, and find that they can be used to shed light on magnitude pruning's performance during the early part of training.