mlsanityData to 7 Oct 2026

VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference

arXiv 2102.04503 ↗First posted 8 Feb 2021cs.LG cs.ARcited by 59

Not on the map. Why?

Abstract (arXiv)

Quantization enables efficient acceleration of deep neural networks by reducing model memory footprint and exploiting low-cost integer math hardware units. Quantization maps floating-point weights and activations in a trained model to low-bitwidth integer values using scale factors. Excessive quantization, reducing precision too aggressively, results in accuracy degradation. When scale factors are shared at a coarse granularity across many dimensions of each tensor, effective precision of individual elements within the tensor are limited. To reduce quantization-related accuracy loss, we propose using a separate scale factor for each small vector of (≈16-64) elements within a single dimension of a tensor. To achieve an efficient hardware implementation, the per-vector scale factors can be implemented with low-bitwidth integers when calibrated using a two-level quantization scheme. We find that per-vector scaling consistently achieves better inference accuracy at low precision compared to conventional scaling techniques for popular neural networks without requiring retraining. We also modify a deep learning accelerator hardware design to study the area and energy overheads of per-vector scaling support. Our evaluation demonstrates that per-vector scaled quantization with 4-bit weights and activations achieves 37% area saving and 24% energy saving while maintaining over 75% accuracy for ResNet50 on ImageNet. 4-bit weights and 8-bit activations achieve near-full-precision accuracy for both BERT-base and BERT-large on SQuAD while reducing area by 26% compared to an 8-bit baseline.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 8 Feb 2021.

Builds on

25 of 40 references.

References · 25

  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  4. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  5. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  6. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  7. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  8. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  9. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  10. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  11. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  12. 2016Trained Ternary Quantizationcited by 203
13 more
  1. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  2. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  3. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  4. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
  5. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  6. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 97
  7. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  8. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  9. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  10. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 43
  11. 2018Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inferencecited by 33
  12. 2017Minimum Energy Quantized Neural Networkscited by 20
  13. 2018Quantization for Rapid Deployment of Deep Neural Networkscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 9; 2024: 19; 2025: 16; 2026: 7.

59 papers.

Sort:

By area:

  1. 2023Microscaling Data Formats for Deep Learningcited by 132
  2. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  5. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  6. 2021Pareto-Optimal Quantized ResNet Is Mostly 4-bitcited by 26
  7. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  8. 2024Instance-Aware Group Quantization for Vision Transformerscited by 23
  9. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  10. 2023Similarity search in the blink of an eye with compressed indicescited by 20
  11. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  12. 2025Recipes for Pre-training LLMs with MXFP8cited by 18
38 more
  1. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  2. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 12
  3. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  4. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 10
  5. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inferencecited by 9
  6. 2023Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?cited by 8
  7. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  8. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inferencecited by 8
  9. 2025MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Servingcited by 8
  10. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 7
  11. 2021TOD: GPU-accelerated Outlier Detection via Tensor Operationscited by 6
  12. 2024Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networkscited by 6
  13. 2025RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remappingcited by 6
  14. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6
  15. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applicationscited by 6
  16. 2024Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsitycited by 5
  17. 2025any4: Learned 4-bit Numeric Representation for LLMscited by 5
  18. 2022Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networkscited by 4
  19. 2022Empirical Evaluation of Post-Training Quantization Methods for Language Taskscited by 4
  20. 2022Hyperspherical Quantization: Toward Smaller and More Accurate Modelscited by 4
  21. 2024FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantizationcited by 4
  22. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  23. 2024PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networkscited by 3
  24. 2024SDQ: Sparse Decomposed Quantization for LLM Inferencecited by 3
  25. 2024Robust Training of Neural Networks at Arbitrary Precision and Sparsitycited by 3
  26. 2025Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoringcited by 3
  27. 2025SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsitycited by 3
  28. 2026Adaptive Block-Scaled Data Typescited by 3
  29. 2022Block Format Error Bounds and Optimal Block Size Selectioncited by 2
  30. 2023Distribution-Flexible Subset Quantization for Post-Quantizing Super-Resolution Networkscited by 2
  31. 2023A Dataflow Compiler for Efficient LLM Inference using Custom Microscaling Formatscited by 2
  32. 2024An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networkscited by 2
  33. 2023INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformerscited by 1
  34. 2024Scaling Laws For Mixed Quantizationcited by 1
  35. 2024Data-Free Group-Wise Fully Quantized Winograd Convolution via Learnable Scalescited by 1
  36. 2025A Segmented Robot Grasping Perception Neural Network for Edge AIcited by 1
  37. 2025SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precisioncited by 1
  38. 2021Variability-Aware Training and Self-Tuning of Highly Quantized DNNs for Analog PIMcited by 0

50 most cited of 59 All 59 (JSON)