# FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer

arXiv: https://arxiv.org/abs/2601.21750 · HTML: https://mlsanity.com/paper/2601.21750/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [Adam-mini: Use Fewer Learning Rates To Gain More](https://mlsanity.com/paper/2406.16793/) (2024)
- [Muon Optimizes Under Spectral Norm Constraints](https://mlsanity.com/paper/2506.15054/) (2025)
- [A Note on the Convergence of Muon](https://mlsanity.com/paper/2502.02900/) (2025)
- [ASGO: Adaptive Structured Gradient Optimization](https://mlsanity.com/paper/2503.20762/) (2025)
- [PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective](https://mlsanity.com/paper/2505.21799/) (2025)
- [Muon Outperforms Adam in Tail-End Associative Memory Learning](https://mlsanity.com/paper/2509.26030/) (2025)
- [Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?](https://mlsanity.com/paper/2511.00674/) (2025)
- [MuonBP: Faster Muon via Block-Periodic Orthogonalization](https://mlsanity.com/paper/2510.16981/) (2025)
- [The Complexity of Large-scale Convex Programming under a Linear Optimization Oracle](https://mlsanity.com/paper/1309.5550/) (2013)
- [Muon Optimizer Accelerates Grokking](https://mlsanity.com/paper/2504.16041/) (2025)
- [Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis](https://mlsanity.com/paper/2303.05473/) (2023)
- [True Asymptotic Natural Gradient Optimization](https://mlsanity.com/paper/1712.08449/) (2017)

## Built on by (most cited first, 11 of 11)

- [The Newton-Muon Optimizer](https://mlsanity.com/paper/2604.01472/) (2026)
- [Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning](https://mlsanity.com/paper/2603.09697/) (2026)
- [MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration](https://mlsanity.com/paper/2603.28254/) (2026)
- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation](https://mlsanity.com/paper/2605.07067/) (2026)
- [Tight Complexity of Muon under Heavy-Tailed Noise](https://mlsanity.com/paper/2605.18528/) (2026)
- [MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning](https://mlsanity.com/paper/2608.05088/) (2026)
- [Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature](https://mlsanity.com/paper/2608.09763/) (2026)
- [Beyond the Matrix Sign: Quadratic Spectral Descent](https://mlsanity.com/paper/2609.07597/) (2026)
- [Rethinking Bregman Divergences in Kronecker-Factored Optimizers](https://mlsanity.com/paper/2606.00542/) (2026)
- [DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon](https://mlsanity.com/paper/2610.06578/) (2026)

## Abstract (arXiv)

Training large-scale neural networks requires solving nonconvex optimization where the choice of optimizer fundamentally determines both convergence behavior and computational efficiency. While adaptive methods like Adam have long dominated practice, the recently proposed Muon optimizer achieves superior performance through orthogonalized momentum updates that enforce isotropic geometry with uniform singular values. However, this strict isotropy discards potentially valuable curvature information encoded in gradient spectra, motivating optimization methods that balance geometric structure with adaptivity. We introduce FISMO (Fisher-Structured Momentum-Orthogonalized) optimizer, which generalizes isotropic updates to incorporate anisotropic curvature information through Fisher information geometry. By reformulating the optimizer update as a trust-region problem constrained by a Kronecker-factored Fisher metric, FISMO achieves structured preconditioning that adapts to local loss landscape geometry while maintaining computational tractability. We establish convergence guarantees for FISMO in stochastic nonconvex settings, proving an O(1/√T) rate for the expected squared gradient norm with explicit characterization of variance reduction through mini-batching. Empirical evaluation on image classification and language modeling benchmarks demonstrates that FISMO achieves superior training efficiency and final performance compared to established baselines.