# Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs

arXiv: https://arxiv.org/abs/2203.07808 · HTML: https://mlsanity.com/paper/2203.07808/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 96 references that are in the map.
- First posted on arXiv 15 Mar 2022.

## Builds on (most cited first, 30 of 62)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)

## Built on by (most cited first, 16 of 16)

- [Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective](https://mlsanity.com/paper/2303.11906/) (2023)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [TULIP: Transformer for Upsampling of LiDAR Point Clouds](https://mlsanity.com/paper/2312.06733/) (2023)
- [Variance-Based Pruning for Accelerating and Compressing Trained Networks](https://mlsanity.com/paper/2507.12988/) (2025)
- [MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protection](https://mlsanity.com/paper/2403.04149/) (2024)
- [Pushing the Efficiency Limit Using Structured Sparse Convolutions](https://mlsanity.com/paper/2210.12818/) (2022)
- [CNN Mixture-of-Depths](https://mlsanity.com/paper/2409.17016/) (2024)
- [PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNs](https://mlsanity.com/paper/2505.03254/) (2025)
- [Weight Compander: A Simple Weight Reparameterization for Regularization](https://mlsanity.com/paper/2306.16993/) (2023)
- [Enhancing Generalization in Convolutional Neural Networks through Regularization with Edge and Line Features](https://mlsanity.com/paper/2410.16897/) (2024)
- [Advancing Weight and Channel Sparsification with Enhanced Saliency](https://mlsanity.com/paper/2502.03658/) (2025)
- [LNPT: Label-free Network Pruning and Training](https://mlsanity.com/paper/2403.12690/) (2024)
- [Convolutional Neural Network Compression Based on Low-Rank Decomposition](https://mlsanity.com/paper/2408.16289/) (2024)
- [Accelerating Inference of Networks in the Frequency Domain](https://mlsanity.com/paper/2410.04342/) (2024)
- [HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks](https://mlsanity.com/paper/2606.30516/) (2026)
- [Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?](https://mlsanity.com/paper/2607.25529/) (2026)

## Abstract (arXiv)

Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs), both at training and inference time. CNNs contain parameters arranged in K × K filters. Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero, thereby specifying a fixed subspace that constrains the filter. Especially if pruning is applied before or during training, this induces a strong bias. To overcome this, we introduce interspace pruning (IP), a general tool to improve existing pruning methods. It uses filters represented in a dynamic interspace by linear combinations of an underlying adaptive filter basis (FB). For IP, FB coefficients are set to zero while un-pruned coefficients and FBs are trained jointly. In this work, we provide mathematical evidence for IP's superior performance and demonstrate that IP outperforms SP on all tested state-of-the-art unstructured pruning methods. Especially in challenging situations, like pruning for ImageNet or pruning to high sparsity, IP greatly exceeds SP with equal runtime and parameter costs. Finally, we show that advances of IP are due to improved trainability and superior generalization ability.