mlsanityData to 7 Oct 2026

u-μP: The Unit-Scaled Maximal Update Parametrization

arXiv 2407.17465 ↗First posted 24 Jul 2024cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The Maximal Update Parametrization (μP) aims to make the optimal hyperparameters (HPs) of a model independent of its size, allowing them to be swept using a cheap proxy model rather than the full-size target model. We present a new scheme, u-μP, which improves upon μP by combining it with Unit Scaling, a method for designing models that makes them easy to train in low-precision. The two techniques have a natural affinity: μP ensures that the scale of activations is independent of model size, and Unit Scaling ensures that activations, weights and gradients begin training with a scale of one. This synthesis opens the door to a simpler scheme, whose default values are near-optimal. This in turn facilitates a more efficient sweeping strategy, with u-μP models reaching a loss that is equal to or lower than comparable μP models and working out-of-the-box in FP8.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 56 of its 85 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

56 of 85 references.

References · 56

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
38 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  2. 2017Mixed Precision Trainingcited by 1,135
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2023Vision Transformers Need Registerscited by 844
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2023The Falcon Series of Open Language Modelscited by 526
  10. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  12. 2024Massive Activations in Large Language Modelscited by 277
  13. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  14. 2022FP8 Formats for Deep Learningcited by 187
  15. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  16. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  17. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  18. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  19. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  20. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  21. 2023Microscaling Data Formats for Deep Learningcited by 132
  22. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  23. 2023A Spectral Condition for Feature Learningcited by 101
  24. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  25. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
  26. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  27. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  28. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  29. 2024Scalable Optimization in the Modular Normcited by 64
  30. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  31. 2023Intriguing Properties of Quantization at Scalecited by 48
  32. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  33. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  34. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  35. 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 35
  36. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
  37. 20228-bit Numerical Formats for Deep Neural Networkscited by 23
  38. 2019Mixed Precision Training With 8-bit Floating Pointcited by 22

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 12.

29 papers.

Sort:

By area:

  1. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  2. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  3. 2024Scaling Optimal LR Across Token Horizonscited by 35
  4. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 30
  5. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  6. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 21
  7. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 21
  8. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  9. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  10. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  11. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 8
  12. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
17 more
  1. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  2. 2025μnit Scaling: Simple and Scalable FP8 LLM Trainingcited by 5
  3. 2025Efficient Hyperparameter Tuning via Trajectory Invariance Principlecited by 4
  4. 2025A Proof of Learning Rate Transfer under μPcited by 3
  5. 2026Hyperball May Not Be a Free Lunchcited by 3
  6. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  7. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  8. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  9. 2025Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operatorscited by 1
  10. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmeticcited by 1
  11. 2026Learning Rate Transfer in Normalized Transformerscited by 1
  12. 2025Re-examining Low Rank adaptation for private LLM fine-tuningcited by 0
  13. 2026Towards a Principled Muon under μmathsfP: Ensuring Spectral Conditions throughout Trainingcited by 0
  14. 2026Extending μP: Spectral Conditions for Feature Learning Across Optimizerscited by 0
  15. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  16. 2026Unlocking Feature Learning in Gated Delta Networks at Scalecited by 0
  17. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0