mlsanityData to 7 Oct 2026

Scaling Laws for Precision

arXiv 2411.04330 ↗First posted 7 Nov 2024cs.LG cs.CLcited by 92

Not on the map. Why?

Abstract (arXiv)

Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision may be compute optimal. We unify the scaling laws for post and pretraining quantization to arrive at a single functional form that predicts degradation from training and inference in varied precisions. We fit on over 465 pretraining runs and validate our predictions on model sizes up to 1.7B parameters trained on up to 26B tokens.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 90 of its 99 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

90 of 99 references.

References · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  13. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  15. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  16. 2017Mixed Precision Trainingcited by 1,135
  17. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  18. 2023StarCoder: may the source be with you!cited by 790
  19. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  20. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  21. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  22. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  23. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  24. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  25. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  26. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  27. 2023Scaling Data-Constrained Language Modelscited by 391
  28. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  29. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  30. 2021Explaining Neural Scaling Lawscited by 353
  31. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  32. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  33. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  34. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  35. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  36. 2023A Survey on Model Compression for Large Language Modelscited by 263
  37. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  38. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 58; 2026: 30.

92 papers.

Sort:

By area:

  1. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  2. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  3. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  4. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  5. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  6. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
  7. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokenscited by 18
  8. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  9. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 16
  10. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  11. 2025Scaling Inference-Efficient Language Modelscited by 14
  12. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
38 more
  1. 2025MetaOpenFOAM 2.0: Large Language Model Driven Chain of Thought for Automating CFD Simulation and Post-Processingcited by 13
  2. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 13
  3. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  4. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  5. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 10
  6. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  7. 2025Model Merging Scaling Laws in Large Language Modelscited by 10
  8. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 9
  9. 2026Prescriptive Scaling Laws for Data Constrained Trainingcited by 9
  10. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  11. 2025Physics of Skill Learningcited by 8
  12. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?cited by 8
  13. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 8
  14. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 7
  15. 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
  16. 2025Automating Structural Engineering Workflows with Large Language Model Agentscited by 6
  17. 2025Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Modelscited by 5
  18. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  19. 2025Improving the Straight-Through Estimator with Zeroth-Order Informationcited by 5
  20. 2026TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Trainingcited by 5
  21. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 5
  22. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?cited by 4
  23. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 4
  24. 2025Characterization and Mitigation of Training Instabilities in Microscaling Formatscited by 4
  25. 2025Compute-Optimal Quantization-Aware Trainingcited by 4
  26. 2025Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theorycited by 4
  27. 2025Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMscited by 4
  28. 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Trainingcited by 4
  29. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 4
  30. 2026Towards Robust Scaling Laws for Optimizerscited by 4
  31. 2025The Race to Efficiency: A New Perspective on AI Scaling Lawscited by 3
  32. 2025LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuningcited by 3
  33. 2025Unified Scaling Laws for Compressed Representationscited by 3
  34. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  35. 2025Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Modelscited by 3
  36. 2025A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamicscited by 3
  37. 2026WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Pointscited by 3
  38. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2

50 most cited of 92 All 92 (JSON)