# A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers

arXiv: https://arxiv.org/abs/1804.03294 · HTML: https://mlsanity.com/paper/1804.03294/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2018.

## Built on by (most cited first, 30 of 120)

- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [Pruning has a disparate impact on model accuracy](https://mlsanity.com/paper/2205.13574/) (2022)
- [PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection](https://mlsanity.com/paper/2205.11098/) (2022)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learning](https://mlsanity.com/paper/2102.03214/) (2021)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [Content-Aware GAN Compression](https://mlsanity.com/paper/2104.02244/) (2021)
- [Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets](https://mlsanity.com/paper/2202.04736/) (2022)
- [Effective Model Sparsification by Scheduled Grow-and-Prune Methods](https://mlsanity.com/paper/2106.09857/) (2021)
- [MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Training](https://mlsanity.com/paper/2312.08656/) (2023)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Single-Shot Pruning for Offline Reinforcement Learning](https://mlsanity.com/paper/2112.15579/) (2021)
- [Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network](https://mlsanity.com/paper/2204.02738/) (2022)
- [Perception-Distortion Balanced ADMM Optimization for Single-Image Super-Resolution](https://mlsanity.com/paper/2208.03324/) (2022)
- [ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models](https://mlsanity.com/paper/2406.07831/) (2024)
- [Towards Sparsification of Graph Neural Networks](https://mlsanity.com/paper/2209.04766/) (2022)
- [A Generic Layer Pruning Method for Signal Modulation Recognition Deep Learning Models](https://mlsanity.com/paper/2406.07929/) (2024)
- [Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off](https://mlsanity.com/paper/2211.16667/) (2022)
- [Binary Complex Neural Network Acceleration on FPGA](https://mlsanity.com/paper/2108.04811/) (2021)
- [Can pruning improve certified robustness of neural networks?](https://mlsanity.com/paper/2206.07311/) (2022)
- [Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression](https://mlsanity.com/paper/2502.16638/) (2025)
- [Pipeline Parallelism for Inference on Heterogeneous Edge Computing](https://mlsanity.com/paper/2110.14895/) (2021)
- [FORMS: Fine-grained Polarized ReRAM-based In-situ Computation for Mixed-signal DNN Accelerator](https://mlsanity.com/paper/2106.09144/) (2021)

## Abstract (arXiv)

Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21 times weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4 times weight reduction in convolutional layers). Our models and codes are released at https://github.com/KaiqiZhang/admm-pruning