mlsanityData to 7 Oct 2026

Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities

arXiv 2505.01043 ↗First posted 2 May 2025cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved impressive performance across various domains. However, the substantial hardware resources required for their training present a significant barrier to efficiency and scalability. To mitigate this challenge, low-precision training techniques have been widely adopted, leading to notable advancements in training efficiency. Despite these gains, low-precision training involves several components, such as weights, activations, and gradients, each of which can be represented in different numerical formats. The resulting diversity has created a fragmented landscape in low-precision training research, making it difficult for researchers to gain a unified overview of the field. This survey provides a comprehensive review of existing low-precision training methods. To systematically organize these approaches, we categorize them into three primary groups based on their underlying numerical formats, which is a key factor influencing hardware compatibility, computational efficiency, and ease of reference for readers. The categories are (1) fixed-point and integer-based methods, (2) floating-point-based methods, and (3) customized format-based methods. Additionally, we discuss quantization-aware training approaches, which share key similarities with low-precision training during forward propagation. Beyond efficiency, we examine robustness and deployment reliability under low precision. Finally, we highlight several promising research directions to advance this field. A collection of papers discussed in this survey is provided in https://github.com/Hao840/Awesome-Low-Precision-Training.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 147 of its 199 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

147 of 199 references.

References · 147

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  13. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  14. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  15. 2019Learned Step Size Quantizationcited by 604
  16. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  17. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  18. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  19. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  20. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  21. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  22. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  23. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  24. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  25. 2015Deep Learning with Limited Numerical Precisioncited by 368
  26. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  27. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  28. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  29. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  30. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  31. 2022FP8 Formats for Deep Learningcited by 187
  32. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  33. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  34. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  35. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  36. 2023Microscaling Data Formats for Deep Learningcited by 132
  37. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  38. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98

50 most cited of 147 All 147 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025Elucidating the Design Space of FP4 trainingcited by 4
  2. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 4
  3. 2026PowLU: An Activation Function for Stable Pre-Training of LLMscited by 4
  4. 2026Stable FP4 Training via Transposition-Invariant Block Quantizationcited by 3
  5. 2026LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantizationcited by 2
  6. 2026Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floorcited by 1
  7. 2026UniSVQ: 2-bit Unified Scalar-Vector Quantizationcited by 1
  8. 2025Reliable Evaluation Protocol for Low-Precision Retrievalcited by 0
  9. 2025Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?cited by 0
  10. 2025FitPro: A Zero-Shot Framework for Interactive Text-based Pedestrian Retrieval in Open Worldcited by 0
  11. 2026Accelerating Sharded Data Parallelism at Scale with Federated Learningcited by 0