# Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation

arXiv: https://arxiv.org/abs/2004.09602 · HTML: https://mlsanity.com/paper/2004.09602/

## Key facts

- Cited by 188 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2020.

## Built on by (most cited first, 30 of 188)

- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021)
- [Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling](https://mlsanity.com/paper/2304.09145/) (2023)
- [Transformers Provably Solve Parity Efficiently with Chain of Thought](https://mlsanity.com/paper/2410.08633/) (2024)
- [Intriguing Properties of Quantization at Scale](https://mlsanity.com/paper/2305.19268/) (2023)
- [FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization](https://mlsanity.com/paper/2306.00317/) (2023)
- [ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats](https://mlsanity.com/paper/2307.09782/) (2023)
- [A Comprehensive Study on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2411.02530/) (2024)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [PTQ4SAM: Post-Training Quantization for Segment Anything](https://mlsanity.com/paper/2405.03144/) (2024)
- [A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms](https://mlsanity.com/paper/2409.16694/) (2024)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production](https://mlsanity.com/paper/2211.10017/) (2022)
- [Pareto-Optimal Quantized ResNet Is Mostly 4-bit](https://mlsanity.com/paper/2105.03536/) (2021)
- [Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training](https://mlsanity.com/paper/2206.06501/) (2022)
- [I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://mlsanity.com/paper/2405.17849/) (2024)
- [Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery](https://mlsanity.com/paper/2601.20088/) (2026)
- [Boost Vision Transformer with GPU-Friendly Sparsity and Quantization](https://mlsanity.com/paper/2305.10727/) (2023)
- [Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models](https://mlsanity.com/paper/2305.12356/) (2023)

## Abstract (arXiv)

Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.