# A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM

arXiv: https://arxiv.org/abs/1811.01907 · HTML: https://mlsanity.com/paper/1811.01907/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 10 of 10)

- [ECNR: Efficient Compressive Neural Representation of Time-Varying Volumetric Datasets](https://mlsanity.com/paper/2311.12831/) (2023)
- [Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression](https://mlsanity.com/paper/2502.16638/) (2025)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [Single-path Bit Sharing for Automatic Loss-aware Model Compression](https://mlsanity.com/paper/2101.04935/) (2021)
- [Dynamic Probabilistic Pruning: A general framework for hardware-constrained pruning at different granularities](https://mlsanity.com/paper/2105.12686/) (2021)
- [FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation](https://mlsanity.com/paper/2203.12091/) (2022)
- [Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding](https://mlsanity.com/paper/2206.15014/) (2022)
- [Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks](https://mlsanity.com/paper/2307.08013/) (2023)
- [Physics Inspired Criterion for Pruning-Quantization Joint Learning](https://mlsanity.com/paper/2312.00851/) (2023)
- [Differentially Private Neural Network Training Under the Hidden State Assumption](https://mlsanity.com/paper/2407.08233/) (2024)

## Abstract (arXiv)

Many model compression techniques of Deep Neural Networks (DNNs) have been investigated, including weight pruning, weight clustering and quantization, etc. Weight pruning leverages the redundancy in the number of weights in DNNs, while weight clustering/quantization leverages the redundancy in the number of bit representations of weights. They can be effectively combined in order to exploit the maximum degree of redundancy. However, there lacks a systematic investigation in literature towards this direction. In this paper, we fill this void and develop a unified, systematic framework of DNN weight pruning and clustering/quantization using Alternating Direction Method of Multipliers (ADMM), a powerful technique in optimization theory to deal with non-convex optimization problems. Both DNN weight pruning and clustering/quantization, as well as their combinations, can be solved in a unified manner. For further performance improvement in this framework, we adopt multiple techniques including iterative weight quantization and retraining, joint weight clustering training and centroid updating, weight clustering retraining, etc. The proposed framework achieves significant improvements both in individual weight pruning and clustering/quantization problems, as well as their combinations. For weight pruning alone, we achieve 167x weight reduction in LeNet-5, 24.7x in AlexNet, and 23.4x in VGGNet, without any accuracy loss. For the combination of DNN weight pruning and clustering/quantization, we achieve 1,910x and 210x storage reduction of weight data on LeNet-5 and AlexNet, respectively, without accuracy loss. Our codes and models are released at the link http://bit.ly/2D3F0np