mlsanityData to 14 Oct 2026

LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

arXiv 2305.17888 ↗First posted 29 May 2023cs.CLcited by 324

Not on the map. Why?

Cited by 34 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

Key facts. Cited by 324 papers in the map (data through 14 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

40 of 47 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
28 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2019Learned Step Size Quantizationcited by 628
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  14. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  15. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  16. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  18. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  19. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  20. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  21. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  22. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  23. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  24. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  25. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  26. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  27. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  28. 2021Data-Free Neural Architecture Search via Recursive Label Calibrationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 33; 2024: 108; 2025: 94; 2026: 89.

324 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2023A Survey on Model Compression for Large Language Modelscited by 269
  7. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  8. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  9. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  10. 2023Efficient Large Language Models: A Surveycited by 186
  11. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  12. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
38 more
  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  2. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  3. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  4. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  5. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  6. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  7. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  8. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  9. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  10. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 100
  11. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  12. 2024Scaling Laws for Precisioncited by 95
  13. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  14. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  15. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  16. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  17. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  18. 2024Evaluating Quantized Large Language Modelscited by 77
  19. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  20. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
  21. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  22. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  23. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  24. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  25. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  26. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  27. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  28. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  29. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  30. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  31. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  32. 2024A Survey on Transformer Compressioncited by 46
  33. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  34. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  35. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  36. 2024Low-Rank Quantization-Aware Training for LLMscited by 43
  37. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  38. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 41

50 most cited of 324 All 324 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
  2. 2025SiLQ: Simple Large Language Model Quantization-Aware Training
  3. 2025DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models
  4. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  5. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?
  6. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  7. 2024Post Training Quantization of Large Language Models with Microscaling Formats
  8. 2024EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs