mlsanityData to 7 Oct 2026

SPFQ: A Stochastic Algorithm and Its Error Analysis for Neural Network Quantization

arXiv 2309.10975 ↗First posted 20 Sep 2023cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantization is a widely used compression method that effectively reduces redundancies in over-parameterized neural networks. However, existing quantization techniques for deep neural networks often lack a comprehensive error analysis due to the presence of non-convex loss functions and nonlinear activations. In this paper, we propose a fast stochastic algorithm for quantizing the weights of fully trained neural networks. Our approach leverages a greedy path-following mechanism in combination with a stochastic quantizer. Its computational complexity scales only linearly with the number of weights in the network, thereby enabling the efficient quantization of large networks. Importantly, we establish, for the first time, full-network error bounds, under an infinite alphabet condition and minimal assumptions on the weights and input data. As an application of this result, we prove that when quantizing a multi-layer network having Gaussian weights, the relative square quantization error exhibits a linear decay as the degree of over-parametrization increases. Furthermore, we demonstrate that it is possible to achieve error bounds equivalent to those obtained in the infinite alphabet case, using on the order of a mere loglog N bits per weight, where N represents the largest number of neurons in a layer.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 42 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

24 of 42 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  10. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  11. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  12. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
12 more
  1. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  2. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  3. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  4. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  5. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  6. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  7. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  8. 2020An Overview of Neural Network Compressioncited by 59
  9. 2022Post-training Quantization for Neural Networks with Provable Guaranteescited by 32
  10. 2020Discrepancy Minimization via a Self-Balancing Walkcited by 22
  11. 2020A Greedy Algorithm for Quantizing Neural Networkscited by 21
  12. 2022A simple approach for quantizing neural networkscited by 7

Built on by

Not cited yet.