mlsanityData to 7 Oct 2026

Distance-aware Quantization

arXiv 2108.06983 ↗First posted 16 Aug 2021cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

We address the problem of network quantization, that is, reducing bit-widths of weights and/or activations to lighten network architectures. Quantization methods use a rounding function to map full-precision values to the nearest quantized ones, but this operation is not differentiable. There are mainly two approaches to training quantized networks with gradient-based optimizers. First, a straight-through estimator (STE) replaces the zero derivative of the rounding with that of an identity function, which causes a gradient mismatch problem. Second, soft quantizers approximate the rounding with continuous functions at training time, and exploit the rounding for quantization at test time. This alleviates the gradient mismatch, but causes a quantizer gap problem. We alleviate both problems in a unified framework. To this end, we introduce a novel quantizer, dubbed a distance-aware quantizer (DAQ), that mainly consists of a distance-aware soft rounding (DASR) and a temperature controller. To alleviate the gradient mismatch problem, DASR approximates the discrete rounding with the kernel soft argmax, which is based on our insight that the quantization can be formulated as a distance-based assignment problem between full-precision values and quantized ones. The controller adjusts the temperature parameter in DASR adaptively according to the input, addressing the quantizer gap problem. Experimental results on standard benchmarks show that DAQ outperforms the state of the art significantly for various bit-widths without bells and whistles.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

46 of 59 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
34 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  4. 2019Learned Step Size Quantizationcited by 604
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  6. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  7. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  8. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  9. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  10. 2016Accelerating the Super-Resolution Convolutional Neural Networkcited by 393
  11. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  12. 2018Model compression via distillation and quantizationcited by 287
  13. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  14. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  15. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  16. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  17. 2016Trained Ternary Quantizationcited by 203
  18. 2016Ternary Weight Networkscited by 195
  19. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
  20. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  21. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  22. 2019Quantization Networkscited by 151
  23. 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 149
  24. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
  25. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  26. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
  27. 2019SFNet: Learning Object-aware Semantic Correspondencecited by 60
  28. 2016Loss-aware Binarization of Deep Networkscited by 56
  29. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  30. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 49
  31. 2015Neural Networks with Few Multiplicationscited by 46
  32. 2017Improving Landmark Localization with Semi-Supervised Learningcited by 39
  33. 2018Loss-aware Weight Quantization of Deep Networkscited by 31
  34. 2016Overcoming Challenges in Fixed Point Training of Deep Convolutional Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 5; 2024: 5; 2025: 3; 2026: 2.

19 papers.

Sort:

By area:

  1. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  2. 2021IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantizationcited by 43
  3. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  4. 2024Instance-Aware Group Quantization for Vision Transformerscited by 23
  5. 2023Quantized Feature Distillation for Network Quantizationcited by 13
  6. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matricescited by 11
  7. 2023EQ-Net: Elastic Quantization Neural Networkscited by 10
  8. 2023MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantizationcited by 6
  9. 2024Custom Gradient Estimators are Straight-Through Estimators in Disguisecited by 6
  10. 2025AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Modelscited by 3
  11. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  12. 2023On Model Explanations with Transferable Neural Pathwayscited by 1
7 more
  1. 2024Scheduling Weight Transitions for Quantization-Aware Trainingcited by 1
  2. 2024Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradientscited by 1
  3. 2023Understanding weight-magnitude hyperparameters in training binary networkscited by 0
  4. 2025Cauchy-Schwarz Regularizerscited by 0
  5. 2025Precision Neural Network Quantization via Learnable Adaptive Modulescited by 0
  6. 2026HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMscited by 0
  7. 2026Neural Network Quantization by Learning Low-Loss Subspacescited by 0