mlsanityData to 7 Oct 2026

FedMuon: Federated Learning with Bias-corrected LMO-based Optimization

arXiv 2509.26337 ↗First posted 30 Sep 2025cs.LG math.OCcited by 7

Not on the map. Why?

Abstract (arXiv)

Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than existing adaptive optimization methods, such as Adam. In this paper, we study how Muon can be utilized in federated learning. We first show that straightforwardly using Muon as the local optimizer of FedAvg does not converge to the stationary point since the LMO is a biased operator. We then propose FedMuon which can mitigate this issue. We also analyze how solving the LMO approximately affects the convergence rate and find that, surprisingly, FedMuon can converge for any number of Newton-Schulz iterations, while it can converge faster as we solve the LMO more accurately. Through experiments, we demonstrated that FedMuon can outperform the state-of-the-art federated learning methods.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

54 of 73 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Group Normalizationcited by 1,514
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2014Convex Optimization: Algorithms and Complexitycited by 569
  11. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  12. 2025Muon is Scalable for LLM Trainingcited by 381
38 more
  1. 2018Sparsified SGD with Memorycited by 373
  2. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  3. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
  4. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  5. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  6. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  7. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  9. 2024The Road Less Scheduledcited by 158
  10. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  11. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 149
  12. 2018D2: Decentralized Training over Decentralized Datacited by 140
  13. 2020Momentum Improves Normalized SGDcited by 136
  14. 2020Optimal Client Sampling for Federated Learningcited by 123
  15. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  16. 2018Communication Compression for Decentralized Trainingcited by 87
  17. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  18. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  19. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  20. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  21. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  22. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  23. 2025Dion: Distributed Orthonormalized Updatescited by 64
  24. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  25. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  26. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 53
  27. 2023Benchmarking Neural Network Training Algorithmscited by 46
  28. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  29. 2022Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reductioncited by 32
  30. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  31. 2022FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcited by 23
  32. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  33. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  34. 2023AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithmscited by 16
  35. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  36. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  37. 2023Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?cited by 14
  38. 2023EControl: Fast Distributed Optimization with Compression and Error Controlcited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  2. 2026CacheMuon: Using Temporal Preconditioning To Approximate Polar Factorcited by 3
  3. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  4. 2026Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedbackcited by 0
  5. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0
  6. 2026FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muoncited by 0
  7. 2026FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Trainingcited by 0