mlsanityData to 14 Oct 2026

A Target-Centric Survey of Quantization-Aware Training

arXiv 2608.29667 ↗First posted 30 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The rapid development of LLMs incurs prohibitive memory footprints and intensive computational demands. Quantization-Aware Training (QAT) techniques have emerged as a promising solution to address these challenges by explicitly simulating quantization effects during model training, yielding low-bit models that achieve accuracy comparable to their full-precision counterparts. In this work, we provide a target-centric survey of QAT, aimed at clarifying both its theoretical foundations and its evolving implementation landscape. We systematically review existing QAT methods through a target-centric taxonomy and synthesize cross-target differences in error characteristics, numerical formats, and strategy transferability. We further summarize QAT evaluation paradigms and discuss challenges in optimization and deployment, outlining potential directions for future research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 65 of its 82 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

65 of 82 references.

References · 65

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
38 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  8. 2021A White Paper on Neural Network Quantizationcited by 472
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  10. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  11. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  12. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  13. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  14. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  15. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  16. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  17. 2016Ternary Weight Networkscited by 204
  18. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  19. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  20. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  21. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  22. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  23. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  24. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  25. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  26. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 96
  27. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  28. 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 90
  29. 2022FP8 Quantization: The Power of the Exponentcited by 88
  30. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  31. 2018Training and Inference with Integers in Deep Neural Networkscited by 86
  32. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 79
  33. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Modelscited by 78
  34. 2024Scaling FP8 training to trillion-token LLMscited by 65
  35. 2023Training Transformers with 4-bit Integerscited by 60
  36. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  37. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  38. 2024Low-Rank Quantization-Aware Training for LLMscited by 43

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
  2. 2025Compute-Optimal Quantization-Aware Training
  3. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  4. 2024Continuous Approximations for Improving Quantization Aware Training of LLMs
  5. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
  6. 20261-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
  7. 2024Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart
  8. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs