mlsanityData to 14 Oct 2026

HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

arXiv 1911.03852 ↗First posted 10 Nov 2019cs.CVcited by 261

Not on the map. Why?

Abstract (arXiv)

Quantization is an effective method for reducing memory footprint and inference time of Neural Networks, e.g., for efficient inference in the cloud, especially at the edge. However, ultra low precision quantization could lead to significant degradation in model generalization. A promising method to address this is to perform mixed-precision quantization, where more sensitive layers are kept at higher precision. However, the search space for a mixed-precision quantization is exponential in the number of layers. Recent work has proposed HAWQ, a novel Hessian based framework, with the aim of reducing this exponential search space by using second-order information. While promising, this prior work has three major limitations: (i) HAWQV1 only uses the top Hessian eigenvalue as a measure of sensitivity and do not consider the rest of the Hessian spectrum; (ii) HAWQV1 approach only provides relative sensitivity of different layers and therefore requires a manual selection of the mixed-precision setting; and (iii) HAWQV1 does not consider mixed-precision activation quantization. Here, we present HAWQV2 which addresses these shortcomings. For (i), we perform a theoretical analysis showing that a better sensitivity metric is to compute the average of all of the Hessian eigenvalues. For (ii), we develop a Pareto frontier based method for selecting the exact bit precision of different layers without any manual selection. For (iii), we extend the Hessian analysis to mixed-precision activation quantization. We have found this to be very beneficial for object detection. We show that HAWQV2 achieves new state-of-the-art results for a wide range of tasks.

Key facts. Cited by 261 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 30; 2023: 49; 2024: 34; 2025: 47; 2026: 69.

261 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  2. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  4. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  5. 2021Post-Training Quantization for Vision Transformercited by 278
  6. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  7. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  8. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  9. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  10. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  11. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  12. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
38 more
  1. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  2. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  3. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 61
  4. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 61
  5. 2023Training Transformers with 4-bit Integerscited by 60
  6. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  7. 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 55
  8. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 54
  9. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 51
  10. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 45
  11. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  12. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 41
  13. 2021OMPQ: Orthogonal Mixed Precision Quantizationcited by 39
  14. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  15. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  16. 2023Diversifying the High-level Features for better Adversarial Transferabilitycited by 34
  17. 2024APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Modelscited by 34
  18. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  19. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  20. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  21. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  22. 2022SDQ: Stochastic Differentiable Quantization with Mixed Precisioncited by 28
  23. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 28
  24. 2026QVLA: Not All Channels Are Equal in Vision-Language-Action Model's Quantizationcited by 28
  25. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  26. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  27. 2024MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Modelscited by 21
  28. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  29. 2021Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomescited by 19
  30. 2024Nearest is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attackscited by 18
  31. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inferencecited by 18
  32. 2023Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learningcited by 17
  33. 2021HAO: Hardware-aware neural Architecture Optimization for Efficient Inferencecited by 16
  34. 2022Wavelet Feature Maps Compression for Image-to-Image CNNscited by 16
  35. 2023CABM: Content-Aware Bit Mapping for Single Image Super-Resolution Network with Large Inputcited by 16
  36. 2023One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Modelscited by 16
  37. 2023Patch-wise Mixed-Precision Quantization of Vision Transformercited by 15
  38. 2021Neural Network Quantization for Efficient Inference: A Surveycited by 14

50 most cited of 261 All 261 (JSON)