# Progressive Weight Pruning of Deep Neural Networks using ADMM

arXiv: https://arxiv.org/abs/1810.07378 · HTML: https://mlsanity.com/paper/1810.07378/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2018.

## Built on by (most cited first, 11 of 11)

- [NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM](https://mlsanity.com/paper/2110.15766/) (2021)
- [ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models](https://mlsanity.com/paper/2406.07831/) (2024)
- [Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach](https://mlsanity.com/paper/2310.16647/) (2023)
- [3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs](https://mlsanity.com/paper/2603.01376/) (2026)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding](https://mlsanity.com/paper/2206.15014/) (2022)
- [PIM-QAT: Neural Network Quantization for Processing-In-Memory (PIM) Systems](https://mlsanity.com/paper/2209.08617/) (2022)
- [Non-transferable Pruning](https://mlsanity.com/paper/2410.08015/) (2024)
- [DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS](https://mlsanity.com/paper/2106.11655/) (2021)
- [DANCE: DAta-Network Co-optimization for Efficient Segmentation Model Training and Inference](https://mlsanity.com/paper/2107.07706/) (2021)
- [P2U: Progressive Precision Update For Efficient Model Distribution](https://mlsanity.com/paper/2506.22871/) (2025)

## Abstract (arXiv)

Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approaches. This work proposes a progressive weight pruning approach based on ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints. Motivated by dynamic programming, the proposed method reaches extremely high pruning rate by using partial prunings with moderate pruning rates. Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios. It achieves up to 34 times pruning rate for ImageNet dataset and 167 times pruning rate for MNIST dataset, significantly higher than those reached by the literature work. Under the same number of epochs, the proposed method also achieves faster convergence and higher compression rates. The codes and pruned DNN models are released in the link bit.ly/2zxdlss