# ARO: A New Lens On Matrix Optimization For Large Models

arXiv: https://arxiv.org/abs/2602.09006 · HTML: https://mlsanity.com/paper/2602.09006/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2026.

## Built on by (most cited first, 7 of 7)

- [The Newton-Muon Optimizer](https://mlsanity.com/paper/2604.01472/) (2026)
- [Muon is Not That Special: Random or Inverted Spectra Work Just as Well](https://mlsanity.com/paper/2605.11181/) (2026)
- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [Tight Complexity of Muon under Heavy-Tailed Noise](https://mlsanity.com/paper/2605.18528/) (2026)
- [Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration](https://mlsanity.com/paper/2605.09034/) (2026)
- [Rotated Manifold Optimization for Low-Rank Adaptation](https://mlsanity.com/paper/2609.34264/) (2026)
- [Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon](https://mlsanity.com/paper/2610.19844/) (2026)

## Abstract (arXiv)

Matrix-based optimizers have attracted growing interest for improving LLM training efficiency, with significant progress centered on orthogonalization/whitening based methods. While yielding substantial performance gains, a fundamental question arises: can we develop new paradigms beyond orthogonalization, pushing the efficiency frontier further? We present Adaptively Rotated Optimization (ARO, a new matrix optimization framework that treats gradient rotation as a first class design principle. ARO accelerates LLM training by performing normed steepest descent in a rotated coordinate system, where the rotation is determined by a novel norm-informed policy. This perspective yields update rules that go beyond existing orthogonalization and whitening optimizers, improving sample efficiency in practice. To make comparisons reliable, we propose a rigorously controlled benchmarking protocol that reduces confounding and bias. Under this protocol, ARO consistently outperforms AdamW (by 1.3 ~1.35×) and orthogonalization methods (by 1.1~1.15×) in LLM pretraining at up to 8B activated parameters, and up to 8× overtrain budget, without evidence of diminishing returns. Finally, we discuss how ARO can be reformulated as a symmetry-aware optimizer grounded in rotational symmetries of residual streams, motivating advanced designs that enable computationally efficient exploitation of cross-layer/cross module couplings.