# Towards the Limit of Network Quantization

arXiv: https://arxiv.org/abs/1612.01543 · HTML: https://mlsanity.com/paper/1612.01543/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2016.

## Built on by (most cited first, 30 of 50)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization](https://mlsanity.com/paper/2211.13609/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://mlsanity.com/paper/2505.07004/) (2025)
- [DKM: Differentiable K-Means Clustering Layer for Neural Network Compression](https://mlsanity.com/paper/2108.12659/) (2021)
- [It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher](https://mlsanity.com/paper/2203.17008/) (2022)
- [The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm](https://mlsanity.com/paper/2507.18553/) (2025)
- [Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes](https://mlsanity.com/paper/2110.13541/) (2021)
- [Stealthy Backdoors as Compression Artifacts](https://mlsanity.com/paper/2104.15129/) (2021)
- [ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNs](https://mlsanity.com/paper/2109.04236/) (2021)
- [HFedMS: Heterogeneous Federated Learning with Memorable Data Semantics in Industrial Metaverse](https://mlsanity.com/paper/2211.03300/) (2022)
- [FIT: A Metric for Model Sensitivity](https://mlsanity.com/paper/2210.08502/) (2022)
- [Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms](https://mlsanity.com/paper/2102.02147/) (2021)
- [Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations](https://mlsanity.com/paper/2110.08271/) (2021)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [Content-Aware Radiance Fields: Aligning Model Complexity with Scene Intricacy Through Learned Bitwidth Quantization](https://mlsanity.com/paper/2410.19483/) (2024)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization](https://mlsanity.com/paper/2112.03690/) (2021)
- [Lossy and Lossless (L2) Post-training Model Size Compression](https://mlsanity.com/paper/2308.04269/) (2023)
- [Model compression as constrained optimization, with application to neural nets. Part V: combining compressions](https://mlsanity.com/paper/2107.04380/) (2021)
- [Effect of Weight Quantization on Learning Models by Typical Case Analysis](https://mlsanity.com/paper/2401.17269/) (2024)
- [ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mamba](https://mlsanity.com/paper/2503.09509/) (2025)
- [Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesis](https://mlsanity.com/paper/2505.20646/) (2025)

## Abstract (arXiv)

Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.