mlsanityData to 7 Oct 2026

Scaling Up Exact Neural Network Compression by ReLU Stability

arXiv 2102.07804 ↗First posted 15 Feb 2021cs.LG math.OCcited by 19

Not on the map. Why?

Abstract (arXiv)

We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 56% of the connections on the CIFAR-10 dataset. The code is available at the following link, https://github.com/yuxwind/ExactCompression.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 61 of its 105 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

61 of 105 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  10. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  11. 2018Rethinking the Value of Network Pruningcited by 595
  12. 2020What is the State of Neural Network Pruning?cited by 578
38 more
  1. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  2. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  4. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  5. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  6. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  7. 2016On the Expressive Power of Deep Neural Networkscited by 302
  8. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  9. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  10. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  11. 2013Predicting Parameters in Deep Learningcited by 223
  12. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  13. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  14. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  15. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  16. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  17. 2019Complexity of Linear Regions in Deep Networkscited by 166
  18. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  19. 2020Characterising Bias in Compressed Modelscited by 162
  20. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  21. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  22. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  23. 2018Strong mixed-integer programming formulations for trained neural networkscited by 115
  24. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  25. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  26. 2017Maximum Resilience of Artificial Neural Networkscited by 97
  27. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  28. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  29. 2015Fast ConvNets Using Group-wise Brain Damagecited by 81
  30. 2019Reverse-Engineering Deep ReLU Networkscited by 81
  31. 2017Verifying Properties of Binarized Deep Neural Networkscited by 77
  32. 2018Training for Faster Adversarial Robustness Verification via Inducing ReLU Stabilitycited by 67
  33. 2020Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routingcited by 61
  34. 2018Wide Compression: Tensor Ring Netscited by 50
  35. 2019Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural networkcited by 39
  36. 2020Lossless Compression of Deep Neural Networkscited by 34
  37. 2020On the Predictability of Pruning Across Scalescited by 33
  38. 2019JANOS: An Integrated Predictive and Prescriptive Modeling Frameworkcited by 27

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 5; 2024: 2; 2025: 1; 2026: 2.

19 papers.

Sort:

  1. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  2. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  3. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  4. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  5. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  6. 2021Parameter identifiability of a deep feedforward ReLU neural networkcited by 24
  7. 2024Optimization Over Trained Neural Networks: Taking a Relaxing Walkcited by 10
  8. 2022Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithmcited by 9
  9. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  10. 2021Universal approximation and model compression for radial neural networkscited by 7
  11. 2023Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networkscited by 6
  12. 2022Quiver neural networkscited by 4
7 more
  1. 2022Training Thinner and Deeper Neural Networks: Jumpstart Regularizationcited by 3
  2. 2022Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networkscited by 1
  3. 2026Relaxation-Informed Training of Neural Network Surrogate Modelscited by 1
  4. 2022Towards Global Neural Network Abstractions with Locally-Exact Reconstructioncited by 0
  5. 2024Tightening convex relaxations of trained neural networks: a unified approach for convex and S-shaped activationscited by 0
  6. 2025Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogatecited by 0
  7. 2026Uncovering the Limits of Proof Sharing for Neural Networkscited by 0