# Muon Optimizer Accelerates Grokking

arXiv: https://arxiv.org/abs/2504.16041 · HTML: https://mlsanity.com/paper/2504.16041/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 6 of 6)

- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Grokking at the Edge of Numerical Stability](https://mlsanity.com/paper/2501.04697/) (2025)

## Built on by (most cited first, 12 of 12)

- [AdaMuon: Adaptive Muon Optimizer](https://mlsanity.com/paper/2507.11005/) (2025)
- [Convergence of Muon with Newton-Schulz](https://mlsanity.com/paper/2601.19156/) (2026)
- [Preconditioning Benefits of Spectral Orthogonalization in Muon](https://mlsanity.com/paper/2601.13474/) (2026)
- [Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking](https://mlsanity.com/paper/2509.21519/) (2025)
- [FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer](https://mlsanity.com/paper/2601.21750/) (2026)
- [Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence](https://mlsanity.com/paper/2605.13079/) (2026)
- [Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization](https://mlsanity.com/paper/2510.00823/) (2025)
- [When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Bounds](https://mlsanity.com/paper/2605.06615/) (2026)
- [Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers](https://mlsanity.com/paper/2608.07436/) (2026)
- [Reassessing Muon for Matrix Factorization](https://mlsanity.com/paper/2607.13246/) (2026)
- [The Active Ingredient in Muon's Grokking](https://mlsanity.com/paper/2607.20512/) (2026)
- [Controlling Polar Exposure to Delay Memorization in Diffusion Models](https://mlsanity.com/paper/2610.02780/) (2026)

## Abstract (arXiv)

This paper investigates the impact of different optimizers on the grokking phenomenon, where models exhibit delayed generalization. We conducted experiments across seven numerical tasks (primarily modular arithmetic) using a modern Transformer architecture. The experimental configuration systematically varied the optimizer (Muon vs. AdamW) and the softmax activation function (standard softmax, stablemax, and sparsemax) to assess their combined effect on learning dynamics. Our empirical evaluation reveals that the Muon optimizer, characterized by its use of spectral norm constraints and second-order information, significantly accelerates the onset of grokking compared to the widely used AdamW optimizer. Specifically, Muon reduced the mean grokking epoch from 153.09 to 102.89 across all configurations, a statistically significant difference (t = 5.0175, p = 6.33e-08). This suggests that the optimizer choice plays a crucial role in facilitating the transition from memorization to generalization.