mlsanityData to 14 Oct 2026

Optimal Scaling Needs Optimal Norm

arXiv 2510.03871 ↗First posted 4 Oct 2025cs.LG cs.AI stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Despite recent progress in optimal hyperparameter transfer under model and dataset scaling, no unifying explanatory principle has been established. For Adam and Scion optimizers, we discover that joint optimal scaling across model and dataset sizes is conditioned on a single invariant: the operator norm of the output layer. Across models with up to 1.3B parameters trained on up to 138B tokens, the optimal learning rate/batch size pair (ηast, Bast) consistently has the same operator norm value - a phenomenon we term norm transfer. This constant norm condition is necessary but not sufficient: while for each dataset size, multiple (η, B) reach the optimal norm, only a unique (ηast, Bast) achieves the best loss. As a sufficient condition, we provide the first measurement of (ηast, Bast) scaling with dataset size for Scion, and find that the scaling rules are consistent with those of Adam. Tuning per-layer-group learning rates also improves model performance, with the output layer being the most sensitive and hidden layers benefiting from lower learning rates. We provide practical insights on norm-guided optimal scaling and release our Distributed Scion (Disco) implementation with logs from over two thousand runs to support research on LLM training dynamics at scale.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

51 of 59 references.

References · 51

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  11. 2018An Empirical Model of Large-Batch Trainingcited by 281
  12. 2024Old Optimizer, New Norm: An Anthologycited by 193
38 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  3. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  4. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  5. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  6. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  7. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  8. 2023A Spectral Condition for Feature Learningcited by 109
  9. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  10. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
  11. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  12. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  13. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  14. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  15. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  16. 2024Modular Duality in Deep Learningcited by 77
  17. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  18. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  19. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  20. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  21. 2024Scalable Optimization in the Modular Normcited by 68
  22. 2025Dion: Distributed Orthonormalized Updatescited by 68
  23. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
  24. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  25. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 57
  26. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  27. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  28. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  29. 2024Scaling Optimal LR Across Token Horizonscited by 40
  30. 2021Batch size-invariance for policy optimizationcited by 35
  31. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 34
  32. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 28
  33. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  34. 2024Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noisecited by 23
  35. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  36. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  37. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  38. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  2. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
  3. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  4. 2026Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networkscited by 2
  5. 2026Towards a Principled Muon under μmathsfP: Ensuring Spectral Conditions throughout Trainingcited by 0
  6. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  7. 2026AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Modelscited by 0
  8. 2026How to scale your HEP ML models: A recipe for robust architecture comparisons at scalecited by 0