# A Physical Response-and-Memory Model for Muon Optimization

arXiv: https://arxiv.org/abs/2608.22994 · HTML: https://mlsanity.com/paper/2608.22994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [An Empirical Model of Large-Batch Training](https://mlsanity.com/paper/1812.06162/) (2018)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [NorMuon: Making Muon more efficient and scalable](https://mlsanity.com/paper/2510.05491/) (2025)
- [The effective noise of Stochastic Gradient Descent](https://mlsanity.com/paper/2112.10852/) (2021)
- [Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum](https://mlsanity.com/paper/2602.17080/) (2026)
- [Demon: Improved Neural Network Training with Momentum Decay](https://mlsanity.com/paper/1910.04952/) (2019)
- [Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization](https://mlsanity.com/paper/2201.11137/) (2022)
- [Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants](https://mlsanity.com/paper/2502.02431/) (2025)
- [Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers](https://mlsanity.com/paper/2307.00631/) (2023)
- [Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning](https://mlsanity.com/paper/2602.16167/) (2026)
- [NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training](https://mlsanity.com/paper/2603.03597/) (2026)

## Abstract (arXiv)

Training large language models is costly. How low a loss the same compute can ultimately reach depends on how each step's gradient is converted into a weight update; the rule that performs this conversion is the optimizer. From SGD and AdamW to the recent Muon, effective update rules have mostly been shaped by engineering intuition and then selected on benchmarks. Muon semi-orthogonalizes the momentum matrix before applying the update and has kept breaking records on public training benchmarks; yet why the semi-orthogonalized direction works, and over how long a history the momentum should average, are two questions at present answered mainly by experience. Here we treat the weight matrix during training as a responsive medium with memory and build a physical model for it, in which both questions find answers: the semi-orthogonalized direction is the maximally dissipative response under an output-side safety budget, which explains why it works; momentum is the internal stress accumulated by the medium; how long it should average is set by the relaxation of this stress, and a real medium relaxes on more than one timescale, the simplest form being one fast and one slow. On this basis we propose the Bi-Maxwell optimizer. The framework further yields a testable consequence: gradient directions change fast early in training and more slowly later, so the optimal memory length should grow with training stage; step-by-step measurements of a proxy for it by a read-only probe across 8 independent training trajectories are consistent with this consequence. Replacing the memory kernel alone, from a single timescale to two, brings training to the target loss in noticeably fewer steps on a public large-language-model optimizer benchmark.