mlsanityData to 7 Oct 2026

Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization

arXiv 2512.09678 ↗First posted 10 Dec 2025math.OC cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this article, we explore the use of various matrix norms for optimizing functions of weight matrices, a crucial problem in deep learning. Moving beyond the spectral norm that underlies the Muon update, we leverage the duals of the Ky Fan norms to introduce the Fanion family of linear minimization oracle (LMO) algorithms, which are closely related to Muon, nu-SAM, and Dion. Staying inside the LMO, we construct the families of F-Fanions and S-Fanions, whose updates are convex combinations of the updates of Fanions and Normalized SGD or SignSGD, respectively. The most promising algorithms in these families are F-Muon and S-Muon. By conducting an extensive empirical study of all three algorithm families across a wide range of tasks and settings, we demonstrate that F-Muon and S-Muon consistently match Muon's performance, while outperforming Muon on a synthetic smooth convex problem.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

27 of 42 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  4. 2025Muon is Scalable for LLM Trainingcited by 381
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  6. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  7. 2024Old Optimizer, New Norm: An Anthologycited by 173
  8. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  9. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  10. 2025A Note on the Convergence of Muoncited by 78
  11. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 75
  12. 2025Practical Efficiency of Muon for Pretrainingcited by 74
15 more
  1. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  2. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  3. 2025Dion: Distributed Orthonormalized Updatescited by 64
  4. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  5. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  6. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  7. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  8. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  9. 2016A Primer on Coordinate Descent Algorithmscited by 16
  10. 2025Error Feedback for Muon and Friendscited by 11
  11. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  12. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  13. 2025Drop-Muon: Update Less, Converge Fastercited by 7
  14. 2025Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimizationcited by 3
  15. 2025Training Neural Networks at Any Scalecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  2. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  3. 2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 1
  4. 2026Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedbackcited by 0