# HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

arXiv: https://arxiv.org/abs/2609.07017 · HTML: https://mlsanity.com/paper/2609.07017/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)
- [Fantastic Pretraining Optimizers and Where to Find Them](https://mlsanity.com/paper/2509.02046/) (2025)
- [Why Do We Need Weight Decay in Modern Deep Learning?](https://mlsanity.com/paper/2310.04415/) (2023)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate](https://mlsanity.com/paper/2010.02916/) (2020)
- [Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks](https://mlsanity.com/paper/2305.17212/) (2023)
- [Controlled LLM Training on Spectral Sphere](https://mlsanity.com/paper/2601.08393/) (2026)
- [Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes](https://mlsanity.com/paper/2209.03695/) (2022)
- [Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization](https://mlsanity.com/paper/2606.16899/) (2026)
- [Rethinking Language Model Scaling under Transferable Hypersphere Optimization](https://mlsanity.com/paper/2603.28743/) (2026)
- [Spherical Perspective on Learning with Normalization Layers](https://mlsanity.com/paper/2006.13382/) (2020)
- [Learning in Compact Spaces with Approximately Normalized Transformer](https://mlsanity.com/paper/2505.22014/) (2025)
- [Improving Deep Learning Optimization through Constrained Parameter Regularization](https://mlsanity.com/paper/2311.09058/) (2023)
- [Hyperball May Not Be a Free Lunch](https://mlsanity.com/paper/2607.22444/) (2026)
- [Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining](https://mlsanity.com/paper/2608.24814/) (2026)
- [Understanding the Disharmony between Weight Normalization Family and Weight Decay: ε-shifted L2 Regularizer](https://mlsanity.com/paper/1911.05920/) (2019)
- [On the Nonlinearity of Learning Rate Scaling for LLM Training](https://mlsanity.com/paper/2606.29158/) (2026)

## Abstract (arXiv)

Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.