WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
Not on the map. Why?
Abstract (arXiv)
Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is its slow convergence and early performance plateau, particularly below 4-bit-widths. While this problem has been observed in prior work, its precise cause remains unclear. In this paper, we analyze the convergence of QAT by estimating the spectrum of the loss-surface Hessians. We find that the weights converge to flat regions around saddle points, where a large fraction of the Hessian eigenvalues are both positive and negative. During training, an increasing fraction of Hessian eigenvalues concentrates around zero, whose magnitude decreases. At lower bit-widths, the magnitude of eigenvalues in the Hessian spectrum is significantly smaller. To mitigate these issues, we propose an algorithm called WinQ to accelerate QAT, which involves: (1) periodically resetting weights to the linear interpolation of full-precision and quantized weights, reducing the distance to the quantization grid and increasing eigenvalue magnitude, and (2) computing gradients of noise-injected weights to regularize the Hessian. Extensive experiments show that WinQ accelerates QAT by up to 4 times across various quantization methods and models. Under the same training cost, WinQ improves state-of-the-art sub-4-bit quantization by up to 8.8%. These results are consistent across 16 settings with different language models, quantization methods, and bit widths.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 17 May 2026.
Builds on
38 of 44 references.
References · 38
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
- 2019Learned Step Size Quantizationcited by 628
- 2024SpinQuant: LLM quantization with learned rotationscited by 471
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
- 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
26 more
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
- 2017How to Escape Saddle Points Efficientlycited by 285
- 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
- 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
- 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
- 2024Scaling Laws for Precisioncited by 95
- 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
- 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 59
- 2025Deep Learning is Not So Mysterious or Differentcited by 56
- 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
- 2024Restructuring Vector Quantization with the Rotation Trickcited by 51
- 2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 40
- 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
- 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
- 2021Analysis of stochastic Lanczos quadrature for spectrum approximationcited by 17
- 2025The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newtoncited by 17
- 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
- 2025Matryoshka Quantizationcited by 14
- 2025Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimationcited by 8
- 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimationcited by 8
- 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizerscited by 6
- 2025LOTION: Smoothing the Optimization Landscape for Quantized Trainingcited by 6
- 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Trainingcited by 4
- 2026Efficient Estimation of Kernel Surrogate Models for Task Attributioncited by 4
- 2025GQ-VAE: A gated quantized VAE for learning variable length tokenscited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025SiLQ: Simple Large Language Model Quantization-Aware Training
- 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models
- 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
- 2026ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models
- 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
- 2026REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
- 2024Post Training Quantization of Large Language Models with Microscaling Formats
- 2024Low-Rank Quantization-Aware Training for LLMs