mlsanityData to 7 Oct 2026

FedMuon: Accelerating Federated Learning with Matrix Orthogonalization

arXiv 2510.27403 ↗First posted 31 Oct 2025cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

The core bottleneck of Federated Learning (FL) lies in the communication rounds. That is, how to achieve more effective local updates is crucial for reducing communication rounds. Existing FL methods still primarily use element-wise local optimizers (Adam/SGD), neglecting the geometric structure of the weight matrices. This often leads to the amplification of pathological directions in the weights during local updates, leading deterioration in the condition number and slow convergence. Therefore, we introduce the Muon optimizer in local, which has matrix orthogonalization to optimize matrix-structured parameters. Experimental results show that, in IID setting, Local Muon significantly accelerates the convergence of FL and reduces communication rounds compared to Local SGD and Local AdamW. However, in non-IID setting, independent matrix orthogonalization based on the local distributions of each client induces strong client drift. Applying Muon in non-IID FL poses significant challenges: (1) client preconditioner leading to client drift; (2) moment reinitialization. To address these challenges, we propose a novel Federated Muon optimizer (FedMuon), which incorporates two key techniques: (1) momentum aggregation, where clients use the aggregated momentum for local initialization; (2) local-global alignment, where the local gradients are aligned with the global update direction to significantly reduce client drift. Theoretically, we prove that textttFedMuon achieves a linear speedup convergence rate without the heterogeneity assumption, where S is the number of participating clients per round, K is the number of local iterations, and R is the total number of communication rounds. Empirically, we validate the effectiveness of FedMuon on language and vision models. Compared to several baselines, FedMuon significantly reduces communication rounds and improves test accuracy.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

20 of 29 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  10. 2019On the Convergence of Adam and Beyondcited by 834
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  12. 2025Muon is Scalable for LLM Trainingcited by 381
8 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  2. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  3. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  4. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  5. 2024Modular Duality in Deep Learningcited by 71
  6. 2024Scalable Optimization in the Modular Normcited by 64
  7. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  8. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 14.

15 papers.

Sort:

By area:

  1. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  2. 2026Improved Convergence Rates of Muon Optimizer for Nonconvex Optimizationcited by 9
  3. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  4. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  5. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 4
  6. 2026DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Modelscited by 1
  7. 2026Deconstructing the Failure of Ideal Noise Correction: A Three-Pillar Diagnosiscited by 1
  8. 2026FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projectioncited by 1
  9. 2026Subspace Optimization for Efficient Federated Learning under Heterogeneous Datacited by 1
  10. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  11. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  12. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0
3 more
  1. 2026CTP-FL: Common-Trajectory Gradient Prediction for Federated Learningcited by 0
  2. 2026FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Trainingcited by 0
  3. 2026FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projectioncited by 0