mlsanityData to 7 Oct 2026

Dynamical Isometry: The Missing Ingredient for Neural Network Pruning

arXiv 2105.05916 ↗First posted 12 May 2021cs.LG cs.AI cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Several recent works [40, 24] observed an interesting phenomenon in neural network pruning: A larger finetuning learning rate can improve the final performance significantly. Unfortunately, the reason behind it remains elusive up to date. This paper is meant to explain it through the lens of dynamical isometry [42]. Specifically, we examine neural network pruning from an unusual perspective: pruning as initialization for finetuning, and ask whether the inherited weights serve as a good initialization for the finetuning? The insights from dynamical isometry suggest a negative answer. Despite its critical role, this issue has not been well-recognized by the community so far. In this paper, we will show the understanding of this problem is very important -- on top of explaining the aforementioned mystery about the larger finetuning rate, it also unveils the mystery about the value of pruning [5, 30]. Besides a clearer theoretical understanding of pruning, resolving the problem can also bring us considerable performance benefits in practice.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

35 of 55 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  12. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
23 more
  1. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2018Rethinking the Value of Network Pruningcited by 595
  4. 2020What is the State of Neural Network Pruning?cited by 578
  5. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  6. 2019The State of Sparsity in Deep Neural Networkscited by 471
  7. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  8. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  9. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  10. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  11. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  12. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  13. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  14. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  15. 2015All you need is a good initcited by 124
  16. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  17. 2020Neural Pruning via Growing Regularizationcited by 97
  18. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  19. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  20. 2015Data-dependent Initializations of Convolutional Neural Networkscited by 29
  21. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  22. 2020WoodFisher: Efficient Second-Order Approximation for Neural Network Compressioncited by 21
  23. 2018Structured Pruning for Efficient ConvNets via Incremental Regularizationcited by 15

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1.

2 papers.

  1. 2022Trainability Preserving Neural Pruningcited by 16
  2. 2024EPSD: Early Pruning with Self-Distillation for Efficient Model Compressioncited by 6