# Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks

arXiv: https://arxiv.org/abs/2308.10438 · HTML: https://mlsanity.com/paper/2308.10438/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Deep Rewiring: Training very sparse deep networks](https://mlsanity.com/paper/1711.05136/) (2017)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [Layer-adaptive sparsity for the Magnitude-based Pruning](https://mlsanity.com/paper/2010.07611/) (2020)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [Adaptive Quantization for Deep Neural Network](https://mlsanity.com/paper/1712.01048/) (2017)
- [Progressive Skeletonization: Trimming more fat from a network at initialization](https://mlsanity.com/paper/2006.09081/) (2020)
- [Effective Sparsification of Neural Networks with Global Sparsity Constraint](https://mlsanity.com/paper/2105.01571/) (2021)
- [Recent Advances on Neural Network Pruning at Initialization](https://mlsanity.com/paper/2103.06460/) (2021)
- [Fighting Quantization Bias With Bias](https://mlsanity.com/paper/1906.03193/) (2019)
- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)

## Built on by (most cited first, 15 of 15)

- [Learning from models beyond fine-tuning](https://mlsanity.com/paper/2310.08184/) (2023)
- [OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization](https://mlsanity.com/paper/2403.12983/) (2024)
- [LPViT: Low-Power Semi-structured Pruning for Vision Transformers](https://mlsanity.com/paper/2407.02068/) (2024)
- [AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment](https://mlsanity.com/paper/2411.10606/) (2024)
- [Layer-Adaptive State Pruning for Deep State Space Models](https://mlsanity.com/paper/2411.02824/) (2024)
- [Visual Detector Compression via Location-Aware Discriminant Analysis](https://mlsanity.com/paper/2509.17968/) (2025)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [DM3D: Distortion-Minimized Weight Pruning for Lossless 3D Object Detection](https://mlsanity.com/paper/2407.02098/) (2024)
- [Electrostatic Force Regularization for Neural Structured Pruning](https://mlsanity.com/paper/2411.11079/) (2024)
- [Hankel Singular Value Regularization for Highly Compressible State Space Models](https://mlsanity.com/paper/2510.22951/) (2025)
- [Channel Fingerprint Construction for Massive MIMO: A Deep Conditional Generative Approach](https://mlsanity.com/paper/2505.07893/) (2025)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [AIRE-Prune: Asymptotic Impulse-Response Energy for State Pruning in State Space Models](https://mlsanity.com/paper/2602.00534/) (2026)
- [Neural expressiveness for beyond importance model compression](https://mlsanity.com/paper/2512.06440/) (2025)
- [Statistical Channel Fingerprint Construction for Massive MIMO: A Unified Tensor Learning Framework](https://mlsanity.com/paper/2604.27574/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel layer-adaptive weight-pruning approach for Deep Neural Networks (DNNs) that addresses the challenge of optimizing the output distortion minimization while adhering to a target pruning ratio constraint. Our approach takes into account the collective influence of all layers to design a layer-adaptive pruning scheme. We discover and utilize a very important additivity property of output distortion caused by pruning weights on multiple layers. This property enables us to formulate the pruning as a combinatorial optimization problem and efficiently solve it through dynamic programming. By decomposing the problem into sub-problems, we achieve linear time complexity, making our optimization algorithm fast and feasible to run on CPUs. Our extensive experiments demonstrate the superiority of our approach over existing methods on the ImageNet and CIFAR-10 datasets. On CIFAR-10, our method achieves remarkable improvements, outperforming others by up to 1.0% for ResNet-32, 0.5% for VGG-16, and 0.7% for DenseNet-121 in terms of top-1 accuracy. On ImageNet, we achieve up to 4.7% and 4.6% higher top-1 accuracy compared to other methods for VGG-16 and ResNet-50, respectively. These results highlight the effectiveness and practicality of our approach for enhancing DNN performance through layer-adaptive weight pruning. Code will be available on https://github.com/Akimoto-Cris/RD_VIT_PRUNE.