mlsanityData to 14 Oct 2026

On Provable Benefits of Muon in Federated Learning

arXiv 2510.03866 ↗First posted 4 Oct 2025cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

The recently introduced optimizer, Muon, has gained increasing attention due to its superior performance across a wide range of applications. However, its effectiveness in federated learning remains unexplored. To address this gap, this paper investigates the performance of Muon in the federated learning setting. Specifically, we propose a new algorithm, FedMuon, and establish its convergence rate for nonconvex problems. Our theoretical analysis reveals multiple favorable properties of FedMuon. In particular, due to its orthonormalized update direction, the learning rate of FedMuon is independent of problem-specific parameters, and, importantly, it can naturally accommodate heavy-tailed noise. The extensive experiments on a variety of neural network architectures validate the effectiveness of the proposed algorithm.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

26 of 37 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  6. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  7. 2025Muon is Scalable for LLM Trainingcited by 428
  8. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  9. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  10. 2024Old Optimizer, New Norm: An Anthologycited by 193
  11. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  12. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
14 more
  1. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  2. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  3. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  4. 2025A Note on the Convergence of Muoncited by 83
  5. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  6. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  7. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  8. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  9. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  10. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  11. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  12. 2022Faster Adaptive Federated Learningcited by 35
  13. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  14. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  2. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  3. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  4. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  5. 2026Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedbackcited by 0
  6. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0
  7. 2026FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muoncited by 0