Data to 14 Oct 2026

Layer-adaptive sparsity for the Magnitude-based Pruning

arXiv 2010.07611 ↗First posted 15 Oct 2020cs.LGcited by 126

Not on the map. Why?

Abstract (arXiv)

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose," the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level ℓ2 distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 19; 2023: 27; 2024: 28; 2025: 24; 2026: 22.

126 papers.

Sort:

By area:

2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
2023DepGraph: Towards Any Structural Pruningcited by 203
2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
2022Advancing Model Pruning via Bi-level Optimizationcited by 56
2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36

38 more

2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
2021GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarizationcited by 29
2021An Information-Theoretic Justification for Model Pruningcited by 25
2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
2024Sparse High Rank Adapterscited by 17
2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
2024LPViT: Low-Power Semi-structured Pruning for Vision Transformerscited by 16
2023A Three-regime Model of Network Pruningcited by 15
2023Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networkscited by 15
2021Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsitycited by 14
2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
2022Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruningcited by 13
2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
2021Prune Your Model Before Distill Itcited by 12
2021How I Learned to Stop Worrying and Love Retrainingcited by 12
2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
2023Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagationcited by 12
2024Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformerscited by 11
2024Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustnesscited by 10
2024Layer-Adaptive State Pruning for Deep State Space Modelscited by 10
2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
2022Theoretical Understanding of the Information Flow on Continual Learning Performancecited by 8
2023My3DGen: A Scalable Personalized 3D Generative Modelcited by 8
2023Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroupcited by 8
2024Towards Meta-Pruning via Optimal Transportcited by 8
2025Efficient Shapley Value-based Non-Uniform Pruning of Large Language Modelscited by 8
2025The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networkscited by 7
2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
2025DLP: Dynamic Layerwise Pruning in Large Language Modelscited by 6
2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
2022OptG: Optimizing Gradient-driven Criteria in Network Sparsitycited by 5
2022Cut Inner Layers: A Structured Pruning Strategy for Efficient U-Net GANscited by 5
2024SNP: Structured Neuron-level Pruning to Preserve Attention Scorescited by 5
2024Holistic Adversarially Robust Pruningcited by 5
2025ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMscited by 5

50 most cited of 126 All 126 (JSON)