# ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers

arXiv: https://arxiv.org/abs/1812.11677 · HTML: https://mlsanity.com/paper/1812.11677/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2018.

## Built on by (most cited first, 30 of 42)

- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search](https://mlsanity.com/paper/2108.08910/) (2021)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets](https://mlsanity.com/paper/2202.04736/) (2022)
- [Effective Model Sparsification by Scheduled Grow-and-Prune Methods](https://mlsanity.com/paper/2106.09857/) (2021)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?](https://mlsanity.com/paper/2102.11068/) (2021)
- [NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM](https://mlsanity.com/paper/2110.15766/) (2021)
- [Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free](https://mlsanity.com/paper/2205.11819/) (2022)
- [FORMS: Fine-grained Polarized ReRAM-based In-situ Computation for Mixed-signal DNN Accelerator](https://mlsanity.com/paper/2106.09144/) (2021)
- [Pruning Adversarially Robust Neural Networks without Adversarial Examples](https://mlsanity.com/paper/2210.04311/) (2022)
- [Improving DNN Fault Tolerance using Weight Pruning and Differential Crossbar Mapping for ReRAM-based Edge AI](https://mlsanity.com/paper/2106.09166/) (2021)
- [GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity](https://mlsanity.com/paper/2108.11033/) (2021)
- [MMVC: Learned Multi-Mode Video Compression with Block-based Prediction Mode Selection and Density-Adaptive Entropy Coding](https://mlsanity.com/paper/2304.02273/) (2023)
- [DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models](https://mlsanity.com/paper/2111.00160/) (2021)
- [Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time Mobile Acceleration](https://mlsanity.com/paper/2111.11581/) (2021)
- [LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification](https://mlsanity.com/paper/2204.02965/) (2022)
- [FPGA Resource-aware Structured Pruning for Real-Time Neural Networks](https://mlsanity.com/paper/2308.05170/) (2023)
- [The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models](https://mlsanity.com/paper/2507.18725/) (2025)
- [CAP-RAM: A Charge-Domain In-Memory Computing 6T-SRAM for Accurate and Precision-Programmable CNN Inference](https://mlsanity.com/paper/2107.02388/) (2021)
- [Neural Network-based OFDM Receiver for Resource Constrained IoT Devices](https://mlsanity.com/paper/2205.06159/) (2022)
- [Variational Nested Dropout](https://mlsanity.com/paper/2101.11353/) (2021)
- [Enabling Fast Deep Learning on Tiny Energy-Harvesting IoT Devices](https://mlsanity.com/paper/2111.14051/) (2021)
- [Speedup deep learning models on GPU by taking advantage of efficient unstructured pruning and bit-width reduction](https://mlsanity.com/paper/2112.15445/) (2021)
- [Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding](https://mlsanity.com/paper/2206.15014/) (2022)
- [Sensi-BERT: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient BERT](https://mlsanity.com/paper/2307.11764/) (2023)
- [Non-transferable Pruning](https://mlsanity.com/paper/2410.08015/) (2024)
- [Compressing Deep Neural Networks Using Explainable AI](https://mlsanity.com/paper/2507.05286/) (2025)

## Abstract (arXiv)

To facilitate efficient embedded and hardware implementations of deep neural networks (DNNs), two important categories of DNN model compression techniques: weight pruning and weight quantization are investigated. The former leverages the redundancy in the number of weights, whereas the latter leverages the redundancy in bit representation of weights. However, there lacks a systematic framework of joint weight pruning and quantization of DNNs, thereby limiting the available model compression ratio. Moreover, the computation reduction, energy efficiency improvement, and hardware performance overhead need to be accounted for besides simply model size reduction. To address these limitations, we present ADMM-NN, the first algorithm-hardware co-optimization framework of DNNs using Alternating Direction Method of Multipliers (ADMM), a powerful technique to deal with non-convex optimization problems with possibly combinatorial constraints. The first part of ADMM-NN is a systematic, joint framework of DNN weight pruning and quantization using ADMM. It can be understood as a smart regularization technique with regularization target dynamically updated in each ADMM iteration, thereby resulting in higher performance in model compression than prior work. The second part is hardware-aware DNN optimizations to facilitate hardware-level implementations. Without accuracy loss, we can achieve 85× and 24× pruning on LeNet-5 and AlexNet models, respectively, significantly higher than prior work. The improvement becomes more significant when focusing on computation reductions. Combining weight pruning and quantization, we achieve 1,910× and 231× reductions in overall model size on these two benchmarks, when focusing on data storage. Highly promising results are also observed on other representative DNNs such as VGGNet and ResNet-50.