# Alternating Direction Method of Multipliers for Quantization

arXiv: https://arxiv.org/abs/2009.03482 · HTML: https://mlsanity.com/paper/2009.03482/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2020.

## Built on by (most cited first, 6 of 6)

- [Synthetic Text Generation for Training Large Language Models via Gradient Matching](https://mlsanity.com/paper/2502.17607/) (2025)
- [SAFE: Finding Sparse and Flat Minima to Improve Pruning](https://mlsanity.com/paper/2506.06866/) (2025)
- [The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM](https://mlsanity.com/paper/2510.01650/) (2025)
- [ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models](https://mlsanity.com/paper/2605.11222/) (2026)
- [NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2602.06694/) (2026)
- [Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework](https://mlsanity.com/paper/2105.09837/) (2021)

## Abstract (arXiv)

Quantization of the parameters of machine learning models, such as deep neural networks, requires solving constrained optimization problems, where the constraint set is formed by the Cartesian product of many simple discrete sets. For such optimization problems, we study the performance of the Alternating Direction Method of Multipliers for Quantization (textttADMM-Q) algorithm, which is a variant of the widely-used ADMM method applied to our discrete optimization problem. We establish the convergence of the iterates of textttADMM-Q to certain textitstationary points. To the best of our knowledge, this is the first analysis of an ADMM-type method for problems with discrete variables/constraints. Based on our theoretical insights, we develop a few variants of textttADMM-Q that can handle inexact update rules, and have improved performance via the use of "soft projection" and "injecting randomness to the algorithm". We empirically evaluate the efficacy of our proposed approaches.