mlsanityData to 7 Oct 2026

Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs

arXiv 2203.07808 ↗First posted 15 Mar 2022cs.CV cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs), both at training and inference time. CNNs contain parameters arranged in K × K filters. Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero, thereby specifying a fixed subspace that constrains the filter. Especially if pruning is applied before or during training, this induces a strong bias. To overcome this, we introduce interspace pruning (IP), a general tool to improve existing pruning methods. It uses filters represented in a dynamic interspace by linear combinations of an underlying adaptive filter basis (FB). For IP, FB coefficients are set to zero while un-pruned coefficients and FBs are trained jointly. In this work, we provide mathematical evidence for IP's superior performance and demonstrate that IP outperforms SP on all tested state-of-the-art unstructured pruning methods. Especially in challenging situations, like pruning for ImageNet or pruning to high sparsity, IP greatly exceeds SP with equal runtime and parameter costs. Finally, we show that advances of IP are due to improved trainability and superior generalization ability.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 62 of its 96 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

62 of 96 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2015Spatial Transformer Networkscited by 1,423
38 more
  1. 2016Group Equivariant Convolutional Networkscited by 1,154
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  5. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  6. 2019Green AIcited by 620
  7. 2020What is the State of Neural Network Pruning?cited by 578
  8. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  10. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  12. 2019The State of Sparsity in Deep Neural Networkscited by 471
  13. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  14. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  15. 2020Meta Pseudo Labelscited by 372
  16. 2020Rethinking Pre-training and Self-trainingcited by 352
  17. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  18. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  19. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  20. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  21. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  22. 2020Scaled-YOLOv4: Scaling Cross Stage Partial Networkcited by 211
  23. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  24. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  25. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  26. 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
  27. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  28. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  29. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  30. 2020Sparse GPU Kernels for Deep Learningcited by 138
  31. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  32. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  33. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  34. 2019Fast Sparse ConvNetscited by 86
  35. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  36. 2020Progressive Skeletonization: Trimming more fat from a network at initializationcited by 64
  37. 2019SpecAugment on Large Scale Datasetscited by 60
  38. 2019Learning Filter Basis for Convolutional Neural Network Compressioncited by 57

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 6; 2025: 3; 2026: 2.

16 papers.

Sort:

  1. 2023Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspectivecited by 17
  2. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  3. 2023TULIP: Transformer for Upsampling of LiDAR Point Cloudscited by 12
  4. 2024MAP: MAsk-Pruning for Source-Free Model Intellectual Property Protectioncited by 4
  5. 2022Pushing the Efficiency Limit Using Structured Sparse Convolutionscited by 3
  6. 2024CNN Mixture-of-Depthscited by 3
  7. 2025Variance-Based Pruning for Accelerating and Compressing Trained Networkscited by 3
  8. 2023Weight Compander: A Simple Weight Reparameterization for Regularizationcited by 1
  9. 2024Enhancing Generalization in Convolutional Neural Networks through Regularization with Edge and Line Featurescited by 1
  10. 2025Advancing Weight and Channel Sparsification with Enhanced Saliencycited by 1
  11. 2025PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNscited by 1
  12. 2024LNPT: Label-free Network Pruning and Trainingcited by 0
4 more
  1. 2024Convolutional Neural Network Compression Based on Low-Rank Decompositioncited by 0
  2. 2024Accelerating Inference of Networks in the Frequency Domaincited by 0
  3. 2026HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networkscited by 0
  4. 2026Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?cited by 0