mlsanityData to 14 Oct 2026

Trainability Preserving Neural Pruning

arXiv 2207.12534 ↗First posted 25 Jul 2022cs.LG cs.AI cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Many recent works have shown trainability plays a central role in neural network pruning -- unattended broken trainability can lead to severe under-performance and unintentionally amplify the effect of retraining learning rate, resulting in biased (or even misinterpreted) benchmark results. This paper introduces trainability preserving pruning (TPP), a scalable method to preserve network trainability against pruning, aiming for improved pruning performance and being more robust to retraining hyper-parameters (e.g., learning rate). Specifically, we propose to penalize the gram matrix of convolutional filters to decorrelate the pruned filters from the retained filters. In addition to the convolutional layers, per the spirit of preserving the trainability of the whole network, we also propose to regularize the batch normalization parameters (scale and bias). Empirical studies on linear MLP networks show that TPP can perform on par with the oracle trainability recovery scheme. On nonlinear ConvNets (ResNet56/VGG19) on CIFAR10/100, TPP outperforms the other counterpart approaches by an obvious margin. Moreover, results on ImageNet-1K with ResNets suggest that TPP consistently performs more favorably against other top-performing structured pruning approaches. Code: https://github.com/MingSun-Tse/TPP.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 51 of its 73 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

51 of 73 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  11. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  12. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
38 more
  1. 2018Rethinking the Value of Network Pruningcited by 599
  2. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  3. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  4. 2019Importance Estimation for Neural Network Pruningcited by 541
  5. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  6. 2019The State of Sparsity in Deep Neural Networkscited by 476
  7. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  8. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  9. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  10. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  11. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 292
  12. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  13. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  14. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  15. 2019Orthogonal Convolutional Neural Networkscited by 123
  16. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  17. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  18. 2020Neural Pruning via Growing Regularizationcited by 97
  19. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  20. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  21. 2015Fast ConvNets Using Group-wise Brain Damagecited by 82
  22. 2020EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruningcited by 80
  23. 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
  24. 2020DMCP: Differentiable Markov Channel Pruning for Neural Networkscited by 69
  25. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  26. 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
  27. 2016ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGAcited by 52
  28. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  29. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  30. 2020A Gradient Flow Framework For Analyzing Network Pruningcited by 36
  31. 2021Towards Compact CNNs via Collaborative Compressioncited by 36
  32. 2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36
  33. 2020Controllable Orthogonalization in Training DNNscited by 30
  34. 2021Locally Free Weight Sharing for Network Width Searchcited by 27
  35. 2022Coarsening the Granularity: Towards Structurally Sparse Lottery Ticketscited by 27
  36. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  37. 2020WoodFisher: Efficient Second-Order Approximation for Neural Network Compressioncited by 21
  38. 2021Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsitycited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 4; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  2. 2023Class-Aware Pruning for Efficient Neural Networkscited by 9
  3. 2023On the Opportunities of Green Computing: A Surveycited by 7
  4. 2025The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Modelscited by 7
  5. 2024Mutual Information Preserving Neural Network Pruningcited by 5
  6. 2026MIS-HCC: Hierarchical Channel Clustering for Efficient Medical Image Segmentationcited by 4
  7. 2024Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinementcited by 3
  8. 2025Boosting Large Language Models with Mask Fine-Tuningcited by 3
  9. 2024pTNAS: Progressive Neural Architecture Search for Tabular Datacited by 2
  10. 2024BiDense: Binarization for Dense Predictioncited by 1
  11. 2023CR-SFP: Learning Consistent Representation for Soft Filter Pruningcited by 0
  12. 2024Subspace Node Pruningcited by 0
4 more
  1. 2025Spatial-Spectral Binarized Neural Network for Panchromatic and Multi-spectral Images Fusioncited by 0
  2. 2025Can abstract concepts from LLM improve SLM performance?cited by 0
  3. 2026Cross-Resolution Diffusion Models via Network Pruningcited by 0
  4. 2026Binarized High-Efficiency RAW Video Restoration and Beyondcited by 0