mlsanityData to 7 Oct 2026

Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

arXiv 2606.16899 ↗First posted 15 Jun 2026cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

51 of 59 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
38 more
  1. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  3. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  4. 2023Llemma: An Open Language Model For Mathematicscited by 385
  5. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  6. 2025Muon is Scalable for LLM Trainingcited by 381
  7. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  8. 2020Query-Key Normalization for Transformerscited by 266
  9. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  10. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  11. 2023A Spectral Condition for Feature Learningcited by 101
  12. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 87
  13. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  14. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  15. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  16. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  17. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  18. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  19. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  20. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  21. 2018Decoupled Networkscited by 45
  22. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  23. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  24. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  25. 2026Controlled LLM Training on Spectral Spherecited by 28
  26. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  27. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 21
  28. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  29. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
  30. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  31. 2025HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalizationcited by 15
  32. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  33. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  34. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 13
  35. 2020Spherical Perspective on Learning with Normalization Layerscited by 9
  36. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  37. 2023Improving Deep Learning Optimization through Constrained Parameter Regularizationcited by 5
  38. 2025Learning in Compact Spaces with Approximately Normalized Transformercited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  2. 2026Hyperball May Not Be a Free Lunchcited by 3
  3. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  4. 2026Scale Weight Decay and Train Bettercited by 2
  5. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 2
  6. 2026RODE: A Radial-Orthogonal Decoupled Engine for Optimizationcited by 0
  7. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  8. 2026Blog: Survey of Optimizerscited by 0
  9. 2026HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperballcited by 0
  10. 2026ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratchcited by 0
  11. 2026Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretrainingcited by 0
  12. 2026Looped Transformers as Optimizerscited by 0