# On the Predictability of Pruning Across Scales

arXiv: https://arxiv.org/abs/2006.10621 · HTML: https://mlsanity.com/paper/2006.10621/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 30 of 33)

- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Scaling Scaling Laws with Board Games](https://mlsanity.com/paper/2104.03113/) (2021)
- [Scaling Laws for Sparsely-Connected Foundation Models](https://mlsanity.com/paper/2309.08520/) (2023)
- [How Do Large Language Monkeys Get Their Power (Laws)?](https://mlsanity.com/paper/2502.17578/) (2025)
- [Scaling Up Exact Neural Network Compression by ReLU Stability](https://mlsanity.com/paper/2102.07804/) (2021)
- [Scaling Laws for Hyperparameter Optimization](https://mlsanity.com/paper/2302.00441/) (2023)
- [Efficient Prediction of Pass@k Scaling in Large Language Models](https://mlsanity.com/paper/2510.05197/) (2025)
- [A Three-regime Model of Network Pruning](https://mlsanity.com/paper/2305.18383/) (2023)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Universality of Winning Tickets: A Renormalization Group Perspective](https://mlsanity.com/paper/2110.03210/) (2021)
- [Reusing Deep Neural Network Models through Model Re-engineering](https://mlsanity.com/paper/2304.00245/) (2023)
- [Modularizing while Training: A New Paradigm for Modularizing DNN Models](https://mlsanity.com/paper/2306.09376/) (2023)
- [A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamics](https://mlsanity.com/paper/2511.07892/) (2025)
- [The Effect of Data Dimensionality on Neural Network Prunability](https://mlsanity.com/paper/2212.00291/) (2022)
- [GPT4Image: Large Pre-trained Models Help Vision Models Learn Better on Perception Task](https://mlsanity.com/paper/2306.00693/) (2023)
- [The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws](https://mlsanity.com/paper/2501.12486/) (2025)
- [Evaluating the Robustness of Chinchilla Compute-Optimal Scaling](https://mlsanity.com/paper/2509.23963/) (2025)
- [Unifying Learning Dynamics and Generalization in Transformers Scaling Law](https://mlsanity.com/paper/2512.22088/) (2025)
- [To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency](https://mlsanity.com/paper/2304.02721/) (2023)
- [Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis](https://mlsanity.com/paper/2308.03128/) (2023)
- [A Step Toward Interpretability: Smearing the Likelihood](https://mlsanity.com/paper/2501.07643/) (2025)
- [Unsupervised Atomic Data Mining via Multi-Kernel Graph Autoencoders for Machine Learning Force Fields](https://mlsanity.com/paper/2509.12358/) (2025)
- [Network Pruning Spaces](https://mlsanity.com/paper/2304.09453/) (2023)
- [Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2306.09863/) (2023)
- [Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning](https://mlsanity.com/paper/2403.15022/) (2024)
- [Non-Gaussianities in Collider Metric Binning](https://mlsanity.com/paper/2503.03809/) (2025)
- [ChromaFormer: A Scalable and Accurate Transformer Architecture for Land Cover Classification](https://mlsanity.com/paper/2503.08534/) (2025)

## Abstract (arXiv)

We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We functionally approximate the error of the pruned networks, showing it is predictable in terms of an invariant tying width, depth, and pruning level, such that networks of vastly different pruned densities are interchangeable. We demonstrate the accuracy of this approximation over orders of magnitude in depth, width, dataset size, and density. We show that the functional form holds (generalizes) for large scale data (e.g., ImageNet) and architectures (e.g., ResNets). As neural networks become ever larger and costlier to train, our findings suggest a framework for reasoning conceptually and analytically about a standard method for unstructured pruning.