# An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants

arXiv: https://arxiv.org/abs/2510.09827 · HTML: https://mlsanity.com/paper/2510.09827/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Training Deep Learning Models with Norm-Constrained LMOs](https://mlsanity.com/paper/2502.07529/) (2025)
- [Stochastic model-based minimization of weakly convex functions](https://mlsanity.com/paper/1803.06523/) (2018)
- [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://mlsanity.com/paper/1705.07774/) (2017)
- [Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence](https://mlsanity.com/paper/2002.10542/) (2020)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization](https://mlsanity.com/paper/2503.12645/) (2025)
- [A Note on the Convergence of Muon](https://mlsanity.com/paper/2502.02900/) (2025)
- [Modular Duality in Deep Learning](https://mlsanity.com/paper/2410.21265/) (2024)
- [Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)](https://mlsanity.com/paper/2505.13416/) (2025)
- [Scalable Optimization in the Modular Norm](https://mlsanity.com/paper/2405.14813/) (2024)
- [Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivity](https://mlsanity.com/paper/1810.05633/) (2018)
- [PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective](https://mlsanity.com/paper/2505.21799/) (2025)
- [Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data](https://mlsanity.com/paper/2502.04664/) (2025)
- [In Search of Adam's Secret Sauce](https://mlsanity.com/paper/2505.21829/) (2025)
- [The importance of better models in stochastic optimization](https://mlsanity.com/paper/1903.08619/) (2019)
- [MoMo: Momentum Models for Adaptive Learning Rates](https://mlsanity.com/paper/2305.07583/) (2023)
- [Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks](https://mlsanity.com/paper/2410.22069/) (2024)
- [A Stochastic Proximal Polyak Step Size](https://mlsanity.com/paper/2301.04935/) (2023)
- [Implicit Parameter-free Online Learning with Truncated Linear Models](https://mlsanity.com/paper/2203.10327/) (2022)
- [A Model-Based Method for Minimizing CVaR and Beyond](https://mlsanity.com/paper/2305.17498/) (2023)

## Built on by (most cited first, 28 of 28)

- [Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?](https://mlsanity.com/paper/2511.00674/) (2025)
- [The Newton-Muon Optimizer](https://mlsanity.com/paper/2604.01472/) (2026)
- [Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum](https://mlsanity.com/paper/2602.17080/) (2026)
- [Muon is Not That Special: Random or Inverted Spectra Work Just as Well](https://mlsanity.com/paper/2605.11181/) (2026)
- [On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer](https://mlsanity.com/paper/2603.09952/) (2026)
- [Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods](https://mlsanity.com/paper/2510.10777/) (2025)
- [Enhancing LLM Training via Spectral Clipping](https://mlsanity.com/paper/2603.14315/) (2026)
- [Muon Does Not Converge on Convex Lipschitz Functions](https://mlsanity.com/paper/2605.08980/) (2026)
- [Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates](https://mlsanity.com/paper/2511.11466/) (2025)
- [Manifold constrained steepest descent for smooth and closed-set optimization](https://mlsanity.com/paper/2601.21487/) (2026)
- [On the Role of Batch Size in Stochastic Conditional Gradient Methods](https://mlsanity.com/paper/2603.21191/) (2026)
- [Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds](https://mlsanity.com/paper/2605.09238/) (2026)
- [Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence](https://mlsanity.com/paper/2605.13079/) (2026)
- [OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biases](https://mlsanity.com/paper/2602.01105/) (2026)
- [Optimistic Dual Averaging Unifies Modern Optimizers](https://mlsanity.com/paper/2605.11172/) (2026)
- [CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor](https://mlsanity.com/paper/2606.16371/) (2026)
- [When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Bounds](https://mlsanity.com/paper/2605.06615/) (2026)
- [Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective](https://mlsanity.com/paper/2606.21514/) (2026)
- [PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer](https://mlsanity.com/paper/2607.17620/) (2026)
- [Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioning](https://mlsanity.com/paper/2601.07326/) (2026)
- [Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss](https://mlsanity.com/paper/2606.06418/) (2026)
- [Optimization Geometrodynamics: Variational Reduction and Interaction Curvature](https://mlsanity.com/paper/2607.06723/) (2026)
- [Beyond the Matrix Sign: Quadratic Spectral Descent](https://mlsanity.com/paper/2609.07597/) (2026)
- [Step-Size Stability in Stochastic Optimization: A Theoretical Perspective](https://mlsanity.com/paper/2602.09842/) (2026)
- [Causal Optimizer Interaction Calculus: Hidden Geometric Relaxation and Identifiable Interventions](https://mlsanity.com/paper/2607.07206/) (2026)
- [Optimizers for Diffusion Models: A Controlled Benchmark](https://mlsanity.com/paper/2609.23055/) (2026)
- [SoftServe: A Scalable Quasi-Newton Method for Deep Learning](https://mlsanity.com/paper/2610.02182/) (2026)
- [Optimizing Large Language Models with Chained LMOs](https://mlsanity.com/paper/2610.10975/) (2026)

## Abstract (arXiv)

To define a steepest descent method over a neural network, we need to choose a norm for each layer, a way to aggregate these norms across layers, and whether to use normalization. We systematically explore different alternatives for aggregating norms across layers, both formalizing existing combinations of Adam and the recently proposed Muon as a type of non-Euclidean gradient descent, and deriving new variants of the Muon optimizer. Through a comprehensive experimental evaluation of the optimizers within our framework, we find that Muon is sensitive to the choice of learning rate, whereas a new variant we call MuonMax is significantly more robust. We then show how to combine any non-Euclidean gradient method with model based momentum (known as Momo). The new Momo variants of Muon are significantly more robust to hyperparameter tuning, and often achieve a better validation score. Thus for new tasks, where the optimal hyperparameters are not known, we advocate for using Momo in combination with MuonMax to save on costly hyperparameter tuning.