mlsanityData to 14 Oct 2026

A Three-regime Model of Network Pruning

arXiv 2305.18383 ↗First posted 28 May 2023stat.ML cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent work has highlighted the complex influence training hyperparameters, e.g., the number of training epochs, can have on the prunability of machine learning models. Perhaps surprisingly, a systematic approach to predict precisely how adjusting a specific hyperparameter will affect prunability remains elusive. To address this gap, we introduce a phenomenological model grounded in the statistical mechanics of learning. Our approach uses temperature-like and load-like parameters to model the impact of neural network (NN) training hyperparameters on pruning performance. A key empirical result we identify is a sharp transition phenomenon: depending on the value of a load-like parameter in the pruned model, increasing the value of a temperature-like parameter in the pre-pruned model may either enhance or impair subsequent pruning performance. Based on this transition, we build a three-regime model by taxonomizing the global structure of the pruned NN loss landscape. Our model reveals that the dichotomous effect of high temperature is associated with transitions between distinct types of global structures in the post-pruned model. Based on our results, we present three case-studies: 1) determining whether to increase or decrease a hyperparameter for improved pruning; 2) selecting the best model to prune from a family of models; and 3) tuning the hyperparameter of the Sharpness Aware Minimization method for better pruning performance.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 33 of its 58 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

33 of 58 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  9. 2018Rethinking the Value of Network Pruningcited by 599
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  11. 2020What is the State of Neural Network Pruning?cited by 583
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
21 more
  1. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  2. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  3. 2018An Empirical Model of Large-Batch Trainingcited by 281
  4. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  5. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  6. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 126
  7. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  8. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  9. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  10. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  11. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  12. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  13. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  14. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  15. 2020On the Predictability of Pruning Across Scalescited by 33
  16. 2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 25
  17. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  18. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  19. 2021Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscited by 19
  20. 2022Evaluating natural language processing models with generalization metrics that do not need access to any training or testing datacited by 19
  21. 2022CrAM: A Compression-Aware Minimizercited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 4; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  2. 2023AutoVP: An Automated Visual Prompting Framework and Benchmarkcited by 29
  3. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  4. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Trainingcited by 20
  5. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  6. 2024MD tree: a model-diagnostic tree grown on loss landscapecited by 3
  7. 2024Evaluating Loss Landscapes from a Topology Perspectivecited by 3
  8. 2025Understanding Flatness in Generative Models: Its Role and Benefitscited by 3
  9. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  10. 2024Visualizing Loss Functions as Topological Landscape Profilescited by 2
  11. 2025CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Modelscited by 1
  12. 2026Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysiscited by 1
3 more
  1. 2026Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimizationcited by 1
  2. 2025Optimize Incompatible Parameters through Compatibility-aware Knowledge Integrationcited by 0
  3. 2026Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifoldscited by 0