# Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory

arXiv: https://arxiv.org/abs/2603.26554 · HTML: https://mlsanity.com/paper/2603.26554/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 62 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws](https://mlsanity.com/paper/2404.05405/) (2024)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Fantastic Pretraining Optimizers and Where to Find Them](https://mlsanity.com/paper/2509.02046/) (2025)
- [Muon Optimizes Under Spectral Norm Constraints](https://mlsanity.com/paper/2506.15054/) (2025)
- [Transformers, parallel computation, and logarithmic depth](https://mlsanity.com/paper/2402.09268/) (2024)
- [Benchmarking Optimizers for Large Language Model Pretraining](https://mlsanity.com/paper/2509.01440/) (2025)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective](https://mlsanity.com/paper/2505.21799/) (2025)
- [Understanding Factual Recall in Transformers via Associative Memories](https://mlsanity.com/paper/2412.06538/) (2024)
- [Muon Outperforms Adam in Tail-End Associative Memory Learning](https://mlsanity.com/paper/2509.26030/) (2025)
- [Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data](https://mlsanity.com/paper/2502.04664/) (2025)
- [Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?](https://mlsanity.com/paper/2511.00674/) (2025)
- [Learning Compositional Functions with Transformers from Easy-to-Hard Data](https://mlsanity.com/paper/2505.23683/) (2025)
- [Convergence of Muon with Newton-Schulz](https://mlsanity.com/paper/2601.19156/) (2026)
- [Structured Preconditioners in Adaptive Optimization: A Unified Analysis](https://mlsanity.com/paper/2503.10537/) (2025)
- [Emergence and scaling laws in SGD learning of shallow neural networks](https://mlsanity.com/paper/2504.19983/) (2025)
- [When do spectral gradient updates help in deep learning?](https://mlsanity.com/paper/2512.04299/) (2025)
- [Preconditioning Benefits of Spectral Orthogonalization in Muon](https://mlsanity.com/paper/2601.13474/) (2026)
- [Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law](https://mlsanity.com/paper/2505.19227/) (2025)
- [How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data](https://mlsanity.com/paper/2510.22980/) (2025)
- [Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2602.08232/) (2026)
- [Muon in Associative Memory Learning: Training Dynamics and Scaling Laws](https://mlsanity.com/paper/2602.05725/) (2026)
- [Insights on Muon from Simple Quadratics](https://mlsanity.com/paper/2602.11948/) (2026)
- [Dimension-adapted Momentum Outscales SGD](https://mlsanity.com/paper/2505.16098/) (2025)

## Built on by (most cited first, 14 of 14)

- [Muon is Not That Special: Random or Inverted Spectra Work Just as Well](https://mlsanity.com/paper/2605.11181/) (2026)
- [Why Muon Outperforms Adam: A Curvature Perspective](https://mlsanity.com/paper/2606.04662/) (2026)
- [Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learning](https://mlsanity.com/paper/2605.05189/) (2026)
- [Phases of Muon: When Muon Eclipses SignSGD](https://mlsanity.com/paper/2605.09552/) (2026)
- [Move on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizer](https://mlsanity.com/paper/2605.23871/) (2026)
- [Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering](https://mlsanity.com/paper/2606.03899/) (2026)
- [Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success](https://mlsanity.com/paper/2606.14560/) (2026)
- [Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective](https://mlsanity.com/paper/2606.21514/) (2026)
- [Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization](https://mlsanity.com/paper/2606.27216/) (2026)
- [Factual recall in linear associative memories: sharp asymptotics and mechanistic insights](https://mlsanity.com/paper/2605.10795/) (2026)
- [SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws](https://mlsanity.com/paper/2609.07868/) (2026)
- [Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization](https://mlsanity.com/paper/2610.02798/) (2026)
- [How Bregman Divergences Shape Shampoo](https://mlsanity.com/paper/2610.08534/) (2026)
- [Optimizing Large Language Models with Chained LMOs](https://mlsanity.com/paper/2610.10975/) (2026)

## Abstract (arXiv)

Spectral optimizers such as Muon have recently shown strong empirical performance in large-scale language model training, but the source and extent of their advantage remain poorly understood. We study this question through the linear associative memory problem, a tractable model for factual recall in transformer-based models. In particular, we go beyond orthogonal embeddings and consider Gaussian inputs and outputs, which allows the number of stored associations to greatly exceed the embedding dimension. Our main result sharply characterizes the recovery rates of one step of Muon, SGD, and Newton's method on the logistic regression loss under a power law frequency distribution. We show that the storage capacity of Muon significantly exceeds that of SGD, and even matches Newton's method while only using first-order information. Moreover, Muon saturates at a larger critical batch size. We further analyze the multi-step dynamics under a thresholded gradient approximation and show that Muon achieves a substantially faster initial recovery rate than SGD, while both methods eventually converge to the information-theoretic limit at comparable speeds. Experiments on synthetic tasks validate the predicted scaling laws. Our analysis provides a quantitative understanding of the signal amplification of spectral preconditioners and lays the groundwork for establishing scaling laws across more practical language modeling tasks and optimizers.