mlsanityData to 14 Oct 2026

Unit Scaling: Out-of-the-Box Low-Precision Training

arXiv 2303.11257 ↗First posted 20 Mar 2023cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

We present unit scaling, a paradigm for designing deep learning models that simplifies the use of low-precision number formats. Training in FP16 or the recently proposed FP8 formats offers substantial efficiency gains, but can lack sufficient range for out-of-the-box training. Unit scaling addresses this by introducing a principled approach to model numerics: seeking unit variance of all weights, activations and gradients at initialisation. Unlike alternative methods, this approach neither requires multiple training runs to find a suitable scale nor has significant computational overhead. We demonstrate the efficacy of unit scaling across a range of models and optimisers. We further show that existing models can be adapted to be unit-scaled, training BERT-Large in FP16 and then FP8 with no degradation in accuracy.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

52 of 72 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2017Mixed Precision Trainingcited by 1,156
  13. 2021Ethical and social risks of harm from Language Modelscited by 966
  14. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  15. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  16. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  17. 2017Self-Normalizing Neural Networkscited by 598
  18. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  19. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  20. 2021A White Paper on Neural Network Quantizationcited by 472
  21. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  22. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  23. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  24. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  25. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  26. 2022FP8 Formats for Deep Learningcited by 196
  27. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  28. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  29. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  30. 2020The Hardware Lotterycited by 160
  31. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  32. 2022FP8 Quantization: The Power of the Exponentcited by 88
  33. 2019Dissecting the Graphcore IPU Architecture via Microbenchmarkingcited by 47
  34. 2016Recurrent Highway Networkscited by 37
  35. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  36. 20228-bit Numerical Formats for Deep Neural Networkscited by 25
  37. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  38. 2018Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seqcited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 6; 2026: 6.

17 papers.

Sort:

By area:

  1. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  2. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 30
  3. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  4. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  5. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  6. 2024eXmY: A Data Type and Technique for Arbitrary Bit Precision Quantizationcited by 7
  7. 2025μnit Scaling: Simple and Scalable FP8 LLM Trainingcited by 5
  8. 2024Scalify: scale propagation for efficient low-precision LLM trainingcited by 4
  9. 2025Elucidating the Design Space of FP4 trainingcited by 4
  10. 2026Learning Rate Transfer in Normalized Transformerscited by 3
  11. 2025MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scalingcited by 2
  12. 2026Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Trainingcited by 2
5 more
  1. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  2. 2025Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Roundingcited by 1
  3. 2026GQA-μP: The maximal parameterization update for grouped query attentioncited by 1
  4. 2026Auditing Invisible Weight Updates with Reference Tracescited by 0
  5. 2026Backward-State Policy Is Part of the Learning Algorithmcited by 0