mlsanityData to 7 Oct 2026

DNN Quantization with Attention

arXiv 2103.13322 ↗First posted 24 Mar 2021cs.CV cs.CCcited by 1

Not on the map. Why?

Abstract (arXiv)

Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). However, low-bit quantization can also cause a considerable drop in accuracy, in particular when we apply it to complex learning tasks or lightweight DNN architectures. In this paper, we propose a training procedure that relaxes the low-bit quantization. We call this procedure textitDNN Quantization with Attention (DQA). The relaxation is achieved by using a learnable linear combination of high, medium and low-bit quantizations. Our learning procedure converges step by step to a low-bit quantization using an attention mechanism with temperature scheduling. In experiments, our approach outperforms other low-bit quantization techniques on various object recognition benchmarks such as CIFAR10, CIFAR100 and ImageNet ILSVRC 2012, achieves almost the same accuracy as a full precision DNN, and considerably reduces the accuracy drop when quantizing lightweight DNN architectures.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

31 of 43 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  9. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  10. 2019Relational Knowledge Distillationcited by 873
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2019Learned Step Size Quantizationcited by 604
19 more
  1. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  2. 2018Born Again Neural Networkscited by 539
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  4. 2015Deep Learning with Limited Numerical Precisioncited by 368
  5. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  6. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  7. 2016Trained Ternary Quantizationcited by 203
  8. 2016Ternary Weight Networkscited by 195
  9. 2014Fractional Max-Poolingcited by 35
  10. 2018BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weightscited by 24
  11. 2019Deep geometric knowledge distillation with graphscited by 21
  12. 2018Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)cited by 19
  13. 2019Mixed Precision DNNs: All you need is a good parametrizationcited by 18
  14. 2020BitPruning: Learning Bitlengths for Aggressive and Accurate Quantizationcited by 14
  15. 2019Differentiable Mask for Pruning Convolutional and Recurrent Networkscited by 10
  16. 2019Iteratively Training Look-Up Tables for Network Quantizationcited by 9
  17. 2019Attention Based Pruning for Shift Networkscited by 7
  18. 2018Quantized Guided Pruning for Efficient Hardware Implementations of Convolutional Neural Networkscited by 5
  19. 2018LIT: Block-wise Intermediate Representation Training for Model Compressioncited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 4