# Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

arXiv: https://arxiv.org/abs/1707.09870 · HTML: https://mlsanity.com/paper/1707.09870/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2017.

## Built on by (most cited first, 30 of 61)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning](https://mlsanity.com/paper/2107.13892/) (2021)
- [Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization](https://mlsanity.com/paper/2208.05163/) (2022)
- [Perception-Distortion Balanced ADMM Optimization for Single-Image Super-Resolution](https://mlsanity.com/paper/2208.03324/) (2022)
- [Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training](https://mlsanity.com/paper/2308.06689/) (2023)
- [SiMaN: Sign-to-Magnitude Network Binarization](https://mlsanity.com/paper/2102.07981/) (2021)
- [Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges](https://mlsanity.com/paper/2309.07438/) (2023)
- [Demystifying and Generalizing BinaryConnect](https://mlsanity.com/paper/2110.13220/) (2021)
- [GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity](https://mlsanity.com/paper/2108.11033/) (2021)
- [Mixed Precision of Quantization of Transformer Language Models for Speech Recognition](https://mlsanity.com/paper/2112.11540/) (2021)
- [Synthetic Text Generation for Training Large Language Models via Gradient Matching](https://mlsanity.com/paper/2502.17607/) (2025)
- [Tequila: Trapping-free Ternary Quantization for Large Language Models](https://mlsanity.com/paper/2509.23809/) (2025)
- [Workflow Optimization for Parallel Split Learning](https://mlsanity.com/paper/2402.10092/) (2024)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions](https://mlsanity.com/paper/2111.00153/) (2021)
- [Quantization Aware Factorization for Deep Neural Network Compression](https://mlsanity.com/paper/2308.04595/) (2023)
- [Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition](https://mlsanity.com/paper/2112.11438/) (2021)
- [Sub-mW Keyword Spotting on an MCU: Analog Binary Feature Extraction and Binary Neural Networks](https://mlsanity.com/paper/2201.03386/) (2022)
- [Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification](https://mlsanity.com/paper/2601.07892/) (2026)
- [Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/2109.03513/) (2021)
- [Deep Neural Networks pruning via the Structured Perspective Regularization](https://mlsanity.com/paper/2206.14056/) (2022)
- [Hyperspherical Quantization: Toward Smaller and More Accurate Models](https://mlsanity.com/paper/2212.12653/) (2022)
- [Learning Discretized Neural Networks under Ricci Flow](https://mlsanity.com/paper/2302.03390/) (2023)
- [Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers](https://mlsanity.com/paper/2407.18175/) (2024)
- [Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems](https://mlsanity.com/paper/2105.01064/) (2021)
- [3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration](https://mlsanity.com/paper/2105.06250/) (2021)

## Abstract (arXiv)

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.