mlsanityData to 7 Oct 2026

DeMuon: A Decentralized Muon for Matrix Optimization over Graphs

arXiv 2510.01377 ↗First posted 1 Oct 2025math.OC cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose DeMuon, a method for decentralized matrix optimization over a given communication topology. DeMuon incorporates matrix orthogonalization via Newton-Schulz iterations-a technique inherited from its centralized predecessor, Muon-and employs gradient tracking to mitigate heterogeneity among local functions. Under heavy-tailed noise conditions and additional mild assumptions, we establish the iteration complexity of DeMuon for reaching an approximate stochastic stationary point. This complexity result matches the best-known complexity bounds of centralized algorithms in terms of dependence on the target tolerance. To the best of our knowledge, DeMuon is the first direct extension of Muon to decentralized optimization over graphs with provable complexity guarantees. We conduct preliminary numerical experiments on decentralized transformer pretraining over graphs with varying degrees of connectivity. Our numerical results demonstrate a clear margin of improvement of DeMuon over other popular decentralized algorithms across different network topologies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

43 of 75 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  4. 2025Muon is Scalable for LLM Trainingcited by 381
  5. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  6. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  7. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  9. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  10. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  11. 2018D2: Decentralized Training over Decentralized Datacited by 140
  12. 2020Momentum Improves Normalized SGDcited by 136
31 more
  1. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  2. 2016NEXT: In-Network Nonconvex Optimizationcited by 107
  3. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
  4. 2018Communication Compression for Decentralized Trainingcited by 87
  5. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  6. 2025A Note on the Convergence of Muoncited by 78
  7. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  8. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  9. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  10. 2025Dion: Distributed Orthonormalized Updatescited by 64
  11. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  12. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  13. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 48
  14. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  15. 2019DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimizationcited by 30
  16. 2014Matrix Completion on Graphscited by 26
  17. 2023High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noisecited by 22
  18. 2019On the Convergence Proof of AMSGrad and a New Versioncited by 21
  19. 2021DeEPCA: Decentralized Exact PCA with Linear Convergence Ratecited by 18
  20. 2021On the Convergence of Decentralized Adaptive Gradient Methodscited by 17
  21. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  22. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  23. 2014Improved Distributed Principal Component Analysiscited by 14
  24. 2025Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noisecited by 11
  25. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  26. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
  27. 2025Decentralized Nonconvex Optimization under Heavy-Tailed Noise: Normalization and Optimal Convergencecited by 5
  28. 2025High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noisecited by 3
  29. 2024In-depth Analysis of Low-rank Matrix Factorisation in a Federated Settingcited by 2
  30. 2025DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Modelscited by 2
  31. 2016A Riemannian gossip approach to decentralized matrix completioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  2. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  3. 2026ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent Systemcited by 1
  4. 2026Schattor: Schatten-family methods for deep learning optimizationcited by 0