# Hyperball May Not Be a Free Lunch

arXiv: https://arxiv.org/abs/2607.22444 · HTML: https://mlsanity.com/paper/2607.22444/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 24 Jul 2026.

## Builds on (most cited first, 22 of 22)

- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [The Road Less Scheduled](https://mlsanity.com/paper/2405.15682/) (2024)
- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)
- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [Three Mechanisms of Weight Decay Regularization](https://mlsanity.com/paper/1810.12281/) (2018)
- [NorMuon: Making Muon more efficient and scalable](https://mlsanity.com/paper/2510.05491/) (2025)
- [Understanding Optimization in Deep Learning with Central Flows](https://mlsanity.com/paper/2410.24206/) (2024)
- [Norm matters: efficient and accurate normalization schemes in deep networks](https://mlsanity.com/paper/1803.01814/) (2018)
- [Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate](https://mlsanity.com/paper/2010.02916/) (2020)
- [Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training](https://mlsanity.com/paper/2505.13738/) (2025)
- [Controlled LLM Training on Spectral Sphere](https://mlsanity.com/paper/2601.08393/) (2026)
- [u-μP: The Unit-Scaled Maximal Update Parametrization](https://mlsanity.com/paper/2407.17465/) (2024)
- [Weight Decay may matter more than muP for Learning Rate Transfer in Practice](https://mlsanity.com/paper/2510.19093/) (2025)
- [Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization](https://mlsanity.com/paper/2606.16899/) (2026)
- [RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization](https://mlsanity.com/paper/2603.20527/) (2026)
- [Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay](https://mlsanity.com/paper/2602.06797/) (2026)
- [ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models](https://mlsanity.com/paper/2605.19095/) (2026)
- [Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws](https://mlsanity.com/paper/2602.14208/) (2026)
- [Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima](https://mlsanity.com/paper/2604.09258/) (2026)
- [OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality](https://mlsanity.com/paper/2606.08783/) (2026)

## Built on by (most cited first, 4 of 4)

- [Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining](https://mlsanity.com/paper/2608.24814/) (2026)
- [Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090](https://mlsanity.com/paper/2608.27370/) (2026)
- [Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining](https://mlsanity.com/paper/2608.28442/) (2026)
- [HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball](https://mlsanity.com/paper/2609.07017/) (2026)

## Abstract (arXiv)

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.