mlsanityData to 14 Oct 2026

StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths

arXiv 2601.19320 ↗First posted 27 Jan 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Quantization-aware training (QAT) is essential for deploying large models under strict memory and latency constraints, yet achieving stable and robust optimization at ultra-low bitwidths remains challenging. Common approaches based on the straight-through estimator (STE) or soft quantizers often suffer from gradient mismatch, instability, or high computational overhead. As such, we propose StableQAT, a unified and efficient QAT framework that stabilizes training in ultra low-bit settings via a novel, lightweight, and theoretically grounded surrogate for backpropagation derived from a discrete Fourier analysis of the rounding operator. StableQAT strictly generalizes STE as the latter arises as a special case of our more expressive surrogate family, yielding smooth, bounded, and inexpensive gradients that improve QAT training performance and stability across various hyperparameter choices. In experiments, StableQAT exhibits stable and efficient QAT at 2-4 bit regimes, demonstrating improved training stability, robustness, and superior performance with negligible training overhead against standard QAT techniques. Our code is available at https://github.com/microsoft/StableQAT.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

35 of 50 references.

References · 35

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2019Learned Step Size Quantizationcited by 628
  10. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  11. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  12. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
23 more
  1. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  2. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  3. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  4. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  5. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  6. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  7. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  8. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  9. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 41
  10. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37
  11. 2022SDQ: Stochastic Differentiable Quantization with Mixed Precisioncited by 28
  12. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  13. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  14. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  15. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  16. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  17. 2025Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compressioncited by 14
  18. 2024SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantizationcited by 6
  19. 2025Beyond Outliers: A Study of Optimizers Under Quantizationcited by 6
  20. 2025Smooth Approximations of the Rounding Functioncited by 3
  21. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformerscited by 2
  22. 2025GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networkscited by 2
  23. 2023MAT: Mixed-Strategy Game of Adversarial Training in Fine-tuningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inferencecited by 0
  2. 2026Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates
  2. 2026A Target-Centric Survey of Quantization-Aware Training
  3. 2025End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
  4. 2026Q-MINO: A Minimal-Norm Method for Quantization-Aware Training
  5. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
  6. 2025Compute-Optimal Quantization-Aware Training
  7. 2022Error-aware Quantization through Noise Tempering
  8. 2026Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin