mlsanityData to 7 Oct 2026

Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networks

arXiv 2410.11203 ↗First posted 15 Oct 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantization reduces the model's hardware costs, such as data movement, storage, and operations like multiply and addition. It also affects the model's behavior by degrading the output quality. Therefore, there is a need for methods that preserve the model's behavior when quantizing model parameters. More exotic numerical encodings, such as block-scaled number formats, have shown advantages for utilizing a fixed bit budget to encode model parameters. This paper presents error diffusion (ED), a hyperparameter-free method for post-training quantization with support for block-scaled data formats. Our approach does not rely on backpropagation or Hessian information. We describe how to improve the quantization process by viewing the neural model as a composite function and diffusing the quantization error in every layer. In addition, we introduce TensorCast, an open-source library based on PyTorch to emulate a variety of number formats, including the block-scaled ones, to aid the research in neural model quantization. We demonstrate the efficacy of our algorithm through rigorous testing on various architectures, including vision and large language models (LLMs), where it consistently delivers competitive results. Our experiments confirm that block-scaled data formats provide a robust choice for post-training quantization and could be used effectively to enhance the practical deployment of advanced neural networks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

30 of 45 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  10. 2021A White Paper on Neural Network Quantizationcited by 453
  11. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  12. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
18 more
  1. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  2. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  3. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  4. 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
  5. 2019Loss Aware Post-training Quantizationcited by 94
  6. 2022FP8 Quantization: The Power of the Exponentcited by 87
  7. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  8. 2019Low-Memory Neural Network Training: A Technical Reportcited by 56
  9. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 49
  10. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  11. 2022Post-training Quantization for Neural Networks with Provable Guaranteescited by 32
  12. 2019Adaptive Loss-aware Quantization for Multi-bit Networkscited by 29
  13. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26
  14. 2023A Cost Analysis of Generative Language Models and Influence Operationscited by 17
  15. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantizationcited by 13
  16. 2019Performance-Efficiency Trade-off of Low-Precision Numerical Formats in Deep Neural Networkscited by 7
  17. 2022Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementationcited by 6
  18. 2023Tailor: Altering Skip Connections for Resource-Efficient Inferencecited by 6

Built on by

Not cited yet.