# Distance-aware Quantization

arXiv: https://arxiv.org/abs/2108.06983 · HTML: https://mlsanity.com/paper/2108.06983/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 59 references that are in the map.
- First posted on arXiv 16 Aug 2021.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Accelerating the Super-Resolution Convolutional Neural Network](https://mlsanity.com/paper/1608.00367/) (2016)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)

## Built on by (most cited first, 19 of 19)

- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization](https://mlsanity.com/paper/2111.09136/) (2021)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Instance-Aware Group Quantization for Vision Transformers](https://mlsanity.com/paper/2404.00928/) (2024)
- [Quantized Feature Distillation for Network Quantization](https://mlsanity.com/paper/2307.10638/) (2023)
- [LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices](https://mlsanity.com/paper/2407.11534/) (2024)
- [EQ-Net: Elastic Quantization Neural Networks](https://mlsanity.com/paper/2308.07650/) (2023)
- [MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization](https://mlsanity.com/paper/2305.08117/) (2023)
- [Custom Gradient Estimators are Straight-Through Estimators in Disguise](https://mlsanity.com/paper/2405.05171/) (2024)
- [AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models](https://mlsanity.com/paper/2510.20348/) (2025)
- [Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Training](https://mlsanity.com/paper/2211.08544/) (2022)
- [Scheduling Weight Transitions for Quantization-Aware Training](https://mlsanity.com/paper/2404.19248/) (2024)
- [On Model Explanations with Transferable Neural Pathways](https://mlsanity.com/paper/2309.09887/) (2023)
- [Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients](https://mlsanity.com/paper/2407.12637/) (2024)
- [Precision Neural Network Quantization via Learnable Adaptive Modules](https://mlsanity.com/paper/2504.17263/) (2025)
- [Understanding weight-magnitude hyperparameters in training binary networks](https://mlsanity.com/paper/2303.02452/) (2023)
- [Cauchy-Schwarz Regularizers](https://mlsanity.com/paper/2503.01639/) (2025)
- [HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs](https://mlsanity.com/paper/2601.20745/) (2026)
- [Neural Network Quantization by Learning Low-Loss Subspaces](https://mlsanity.com/paper/2606.25087/) (2026)

## Abstract (arXiv)

We address the problem of network quantization, that is, reducing bit-widths of weights and/or activations to lighten network architectures. Quantization methods use a rounding function to map full-precision values to the nearest quantized ones, but this operation is not differentiable. There are mainly two approaches to training quantized networks with gradient-based optimizers. First, a straight-through estimator (STE) replaces the zero derivative of the rounding with that of an identity function, which causes a gradient mismatch problem. Second, soft quantizers approximate the rounding with continuous functions at training time, and exploit the rounding for quantization at test time. This alleviates the gradient mismatch, but causes a quantizer gap problem. We alleviate both problems in a unified framework. To this end, we introduce a novel quantizer, dubbed a distance-aware quantizer (DAQ), that mainly consists of a distance-aware soft rounding (DASR) and a temperature controller. To alleviate the gradient mismatch problem, DASR approximates the discrete rounding with the kernel soft argmax, which is based on our insight that the quantization can be formulated as a distance-based assignment problem between full-precision values and quantized ones. The controller adjusts the temperature parameter in DASR adaptively according to the input, addressing the quantizer gap problem. Experimental results on standard benchmarks show that DAQ outperforms the state of the art significantly for various bit-widths without bells and whistles.