# Loss-aware Weight Quantization of Deep Networks

arXiv: https://arxiv.org/abs/1802.08635 · HTML: https://mlsanity.com/paper/1802.08635/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2018.

## Built on by (most cited first, 30 of 31)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs](https://mlsanity.com/paper/2406.01721/) (2024)
- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Compression of Generative Pre-trained Language Models via Quantization](https://mlsanity.com/paper/2203.10705/) (2022)
- [Towards Efficient Post-training Quantization of Pre-trained Language Models](https://mlsanity.com/paper/2109.15082/) (2021)
- [CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution](https://mlsanity.com/paper/2207.10345/) (2022)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization](https://mlsanity.com/paper/2203.11239/) (2022)
- [EcoFormer: Energy-Saving Attention with Linear Complexity](https://mlsanity.com/paper/2209.09004/) (2022)
- [Distance-aware Quantization](https://mlsanity.com/paper/2108.06983/) (2021)
- [Exploring Extreme Parameter Compression for Pre-trained Language Models](https://mlsanity.com/paper/2205.10036/) (2022)
- [ORANSight-2.0: Foundational LLMs for O-RAN](https://mlsanity.com/paper/2503.05200/) (2025)
- [QReg: On Regularization Effects of Quantization](https://mlsanity.com/paper/2206.12372/) (2022)
- [Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction](https://mlsanity.com/paper/2212.05735/) (2022)
- [Class-based Quantization for Neural Networks](https://mlsanity.com/paper/2211.14928/) (2022)
- [Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs](https://mlsanity.com/paper/2107.13490/) (2021)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks](https://mlsanity.com/paper/2307.13337/) (2023)
- [Radio: Rate-Distortion Optimization for Large Language Model Compression](https://mlsanity.com/paper/2505.03031/) (2025)
- [The Binary Quantized Neural Network for Dense Prediction via Specially Designed Upsampling and Attention](https://mlsanity.com/paper/2405.17776/) (2024)
- [FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation](https://mlsanity.com/paper/2203.12091/) (2022)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Ternary Quantization: A Survey](https://mlsanity.com/paper/2303.01505/) (2023)
- [One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training](https://mlsanity.com/paper/2401.16760/) (2024)
- [Foundations of Large Language Model Compression -- Part 1: Weight Quantization](https://mlsanity.com/paper/2409.02026/) (2024)
- [NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization](https://mlsanity.com/paper/2104.13818/) (2021)
- [A Closer Look at Hardware-Friendly Weight Quantization](https://mlsanity.com/paper/2210.03671/) (2022)
- [Hyperspherical Loss-Aware Ternary Quantization](https://mlsanity.com/paper/2212.12649/) (2022)
- [Learning to Learn with Indispensable Connections](https://mlsanity.com/paper/2304.02862/) (2023)

## Abstract (arXiv)

The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization. Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.