mlsanityData to 7 Oct 2026

Quantizing deep convolutional networks for efficient inference: A whitepaper

arXiv 1806.08342 ↗First posted 21 Jun 2018cs.LG cs.CV stat.MLcited by 507

Not on the map. Why?

Abstract (arXiv)

We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.

Key facts. Cited by 507 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 81; 2022: 84; 2023: 89; 2024: 89; 2025: 95; 2026: 69.

507 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021A White Paper on Neural Network Quantizationcited by 453
  3. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  5. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  6. 2021NeRV: Neural Representations for Videoscited by 255
  7. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  9. 2021COIN: COmpression with Implicit Neural representationscited by 232
  10. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  11. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  12. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
38 more
  1. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  2. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  3. 2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 127
  4. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  5. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  6. 2023HNeRV: A Hybrid Neural Representation for Videoscited by 100
  7. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  8. 2023Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarkscited by 90
  9. 2022FP8 Quantization: The Power of the Exponentcited by 87
  10. 2022COIN++: Neural Compression Across Modalitiescited by 81
  11. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Modelscited by 77
  12. 2024Evaluating Quantized Large Language Modelscited by 75
  13. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  14. 2022Compressing Volumetric Radiance Fields to 1 MBcited by 66
  15. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  16. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  17. 2021Implicit Neural Video Compressioncited by 52
  18. 2022Online Continual Learning for Embedded Devicescited by 48
  19. 2023Intriguing Properties of Quantization at Scalecited by 48
  20. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  21. 2021Rethinking "Batch" in BatchNormcited by 44
  22. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  23. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
  24. 2021IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantizationcited by 43
  25. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  26. 2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 42
  27. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  28. 2022Hardware Approximate Techniques for Deep Neural Network Accelerators: A Surveycited by 39
  29. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  30. 2021Torch.fx: Practical Program Capture and Transformation for Deep Learning in Pythoncited by 38
  31. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  32. 2021Q-SpiNN: A Framework for Quantizing Spiking Neural Networkscited by 33
  33. 2022Post-training Quantization for Neural Networks with Provable Guaranteescited by 32
  34. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  35. 2024On the Challenges and Opportunities in Generative AIcited by 31
  36. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  37. 2021Auto-Split: A General Framework of Collaborative Edge-Cloud AIcited by 29
  38. 2021GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarizationcited by 29

50 most cited of 507 All 507 (JSON)