mlsanityData to 14 Oct 2026

Mixed-Precision Quantization for Language Models: Techniques and Prospects

arXiv 2510.16805 ↗First posted 19 Oct 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The rapid scaling of language models (LMs) has resulted in unprecedented computational, memory, and energy requirements, making their training and deployment increasingly unsustainable. Quantization has emerged as an essential compression technique to reduce model size, alleviate memory bottlenecks, and accelerate inference. However, while uniform low-bit quantization (e.g., INT8, INT4) provides significant efficiency gains, it can degrade accuracy in sensitive components of transformer-based LMs. Mixed-precision quantization offers a promising alternative by selectively allocating precision across layers or within tensors to balance efficiency and accuracy. This survey provides a comprehensive overview of Mixed-Precision quantization frameworks for LMs (MXPLMs). We first review quantization fundamentals, including uniform and non-uniform quantizers, quantization granularity, and methods widely used in post-training quantization. We then categorize and compare recent MXPLM frameworks according to their bit allocation strategies and precision configurations across weights, activations, and key-value caches. A comparative analysis highlights differences in perplexity, zero-shot task performance, and deployment trade-offs. Furthermore, we contrast MXPLMs with earlier mixed-precision quantization methods for deep neural networks, identifying strategies that transfer and those that face challenges in the LM setting. Finally, we summarize open issues and future directions, including hardware-aware design, activation quantization, and scalable optimization methods for billion-parameter models. By consolidating recent advances, this work serves as a reference for understanding the current landscape and research prospects of mixed-precision quantization for large-scale language models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 106 of its 181 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

106 of 181 references.

References · 106

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
38 more
  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2021A White Paper on Neural Network Quantizationcited by 472
  12. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  13. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  14. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  15. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  16. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  17. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  18. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  19. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  20. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  21. 2023A Survey on Model Compression for Large Language Modelscited by 269
  22. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  23. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  24. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  25. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  26. 2022FP8 Formats for Deep Learningcited by 196
  27. 2023Efficient Large Language Models: A Surveycited by 186
  28. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  29. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  30. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  31. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  32. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  33. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
  34. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  35. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  36. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  37. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  38. 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 82

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMscited by 3
  2. 2026Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0cited by 0
  3. 2026FlexPosit: Tunable Fractional Precision for LLM Inference Acceleratorscited by 0
  4. 2026From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformerscited by 0
  5. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models
  2. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
  3. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  4. 2026MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
  5. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
  6. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Models
  7. 2024Post Training Quantization of Large Language Models with Microscaling Formats
  8. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels