mlsanityData to 14 Oct 2026

Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions

arXiv 2301.07966 ↗First posted 19 Jan 2023cs.LG math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network, and consequently reduces the expected maximum number of linear regions based on the architecture. We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions and our proposed bound for the maximum number. Conversely, we find out that selecting the sparsity across layers to maximize our bound very often improves accuracy in comparison to pruning as much with the same sparsity in all layers, thereby providing us guidance on where to prune.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 53 of its 81 references that are in the map. First posted on arXiv 19 Jan 2023.

Builds on

53 of 81 references.

References · 53

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  8. 2020What is the State of Neural Network Pruning?cited by 583
  9. 2016Error bounds for approximations with deep ReLU networkscited by 574
  10. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
38 more
  1. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  2. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  3. 2016On the Expressive Power of Deep Neural Networkscited by 306
  4. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  5. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  6. 2013Predicting Parameters in Deep Learningcited by 224
  7. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 221
  8. 2013Maxout Networkscited by 187
  9. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  10. 2019What Do Compressed Deep Neural Networks Forget?cited by 171
  11. 2019Complexity of Linear Regions in Deep Networkscited by 167
  12. 2020Characterising Bias in Compressed Modelscited by 167
  13. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  14. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  15. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 122
  16. 2018Strong mixed-integer programming formulations for trained neural networkscited by 115
  17. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  18. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  19. 2017Maximum Resilience of Artificial Neural Networkscited by 97
  20. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  21. 2015Representation Benefits of Deep Feedforward Networkscited by 86
  22. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  23. 2015Fast ConvNets Using Group-wise Brain Damagecited by 82
  24. 2018Training for Faster Adversarial Robustness Verification via Inducing ReLU Stabilitycited by 67
  25. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  26. 2020Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routingcited by 61
  27. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  28. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  29. 2022Pruning has a disparate impact on model accuracycited by 44
  30. 2018Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Boundscited by 43
  31. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  32. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  33. 2020Lossless Compression of Deep Neural Networkscited by 34
  34. 2019JANOS: An Integrated Predictive and Prescriptive Modeling Frameworkcited by 27
  35. 2018Empirical Bounds on Linear Regions of Deep Rectifier Networkscited by 23
  36. 2020Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruningcited by 23
  37. 2020Prune Responsiblycited by 20
  38. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 3.

8 papers.

  1. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  2. 2024Optimization Over Trained Neural Networks: Taking a Relaxing Walkcited by 10
  3. 2024Multi-task Representation Learning for Mixed Integer Linear Programmingcited by 7
  4. 2025The Computational Complexity of Counting Linear Regions in ReLU Neural Networkscited by 7
  5. 2023Computational Tradeoffs of Optimization-Based Bound Tightening in ReLU Networkscited by 6
  6. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  7. 2024Tightening convex relaxations of trained neural networks: a unified approach for convex and S-shaped activationscited by 0
  8. 2025Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogatecited by 0