mlsanityData to 14 Oct 2026

HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision

arXiv 1905.03696 ↗First posted 29 Apr 2019cs.CVcited by 402

Not on the map. Why?

Abstract (arXiv)

Model size and inference speed/power have become a major challenge in the deployment of Neural Networks for many applications. A promising approach to address these problems is quantization. However, uniformly quantizing a model to ultra low precision leads to significant accuracy degradation. A novel solution for this is to use mixed-precision quantization, as some parts of the network may allow lower precision as compared to other layers. However, there is no systematic way to determine the precision of different layers. A brute force approach is not feasible for deep networks, as the search space for mixed-precision is exponential in the number of layers. Another challenge is a similar factorial complexity for determining block-wise fine-tuning order when quantizing the model to a target precision. Here, we introduce Hessian AWare Quantization (HAWQ), a novel second-order quantization method to address these problems. HAWQ allows for the automatic selection of the relative quantization precision of each layer, based on the layer's Hessian spectrum. Moreover, HAWQ provides a deterministic fine-tuning order for quantizing layers, based on second-order information. We show the results of our method on Cifar-10 using ResNet20, and on ImageNet using Inception-V3, ResNet50 and SqueezeNext models. Comparing HAWQ with state-of-the-art shows that we can achieve similar/better accuracy with 8× activation compression ratio on ResNet20, as compared to DNAS~citewu2018mixed, and up to 1% higher accuracy with up to 14% smaller models on ResNet50 and Inception-V3, compared to recently proposed methods of RVQuant~citepark2018value and HAQ~citewang2018haq. Furthermore, we show that we can quantize SqueezeNext to just 1MB model size while achieving above 68% top1 accuracy on ImageNet.

Key facts. Cited by 402 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 52; 2022: 50; 2023: 74; 2024: 65; 2025: 74; 2026: 87.

402 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  5. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  6. 2021A White Paper on Neural Network Quantizationcited by 472
  7. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  10. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  11. 2023A Survey on Model Compression for Large Language Modelscited by 269
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 255
38 more
  1. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 170
  2. 2021Implicit Neural Representations for Image Compressioncited by 162
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  4. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  5. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  6. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  7. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  8. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  9. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  10. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  11. 2022FP8 Quantization: The Power of the Exponentcited by 88
  12. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  13. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  14. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  15. 2022When Do Flat Minima Optimizers Work?cited by 78
  16. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 70
  17. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
  18. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  19. 2021hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devicescited by 62
  20. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 61
  21. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 61
  22. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  23. 2023Training Transformers with 4-bit Integerscited by 60
  24. 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 55
  25. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 54
  26. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  27. 2023Pruning vs Quantization: Which is Better?cited by 46
  28. 2024A Survey on Transformer Compressioncited by 46
  29. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 45
  30. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  31. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  32. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantizationcited by 43
  33. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  34. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 39
  35. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  36. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36
  37. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  38. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35

50 most cited of 402 All 402 (JSON)