mlsanityData to 7 Oct 2026

A Note on Stability for Orthogonalized Matrix Momentum with Client Sampling

arXiv 2606.01720 ↗First posted 1 Jun 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We study finite-sample generalization for a client-sampled distributed optimization scheme with matrix-valued parameters and orthogonalized momentum updates. The central quantity is the gap between the population and empirical objectives at the returned model when only a subset of clients participates in each round. Under independent heterogeneous client data, unequal local sample counts, and fixed aggregation weights, we derive a finite-round upper-tail guarantee from a coupled-neighbor stability recursion and a weighted concentration step. The bound keeps the client-selection counts through the amplification factor (Yi(mathcal C)); in the uniform full-participation full-batch regime, it yields (widetildemathcal O(n-1+n-1/2)) scaling whenever the horizon-dependent amplification terms are controlled. The matrix-orthogonalization rule is required to be Lipschitz along paired trajectories, a condition satisfied by regularized polar-type maps and normalized finite-step Newton--Schulz orthogonalizers. For the unregularized matrix sign, the same argument requires coupled spectral separation, whereas Gaussian smoothing gives a finite-round smoothed variant. A one-dimensional counterexample shows why a gap, smoothing, or regularity condition is necessary.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

39 of 51 references.

References · 39

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2020Adaptive Federated Optimizationcited by 1,036
  3. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  5. 2025Muon is Scalable for LLM Trainingcited by 381
  6. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  7. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  9. 2024Old Optimizer, New Norm: An Anthologycited by 173
  10. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  11. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
  12. 2019Sharper bounds for uniformly stable algorithmscited by 91
27 more
  1. 2025Dion: Distributed Orthonormalized Updatescited by 64
  2. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  3. 2018Generalization Bounds for Uniformly Stable Algorithmscited by 56
  4. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  5. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  6. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  7. 2026Convergence of Muon with Newton-Schulzcited by 30
  8. 2025On the Convergence of Muon and Beyondcited by 28
  9. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  10. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  11. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  12. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  13. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  14. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 15
  15. 2025Robust Machine Unlearning for Quantized Neural Networks via Adaptive Gradient Reweighting with Similar Labelscited by 12
  16. 2025InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimizationcited by 11
  17. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  18. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
  19. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  20. 2026Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimizationcited by 7
  21. 2025Optimal Complexity in Byzantine-Robust Distributed Stochastic Optimization with Data Heterogeneitycited by 6
  22. 2026TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Trainingcited by 5
  23. 2025Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFTcited by 4
  24. 2026Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergencecited by 4
  25. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoningcited by 4
  26. 2026Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networkscited by 2
  27. 2026MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Muse: Representation Geometry of Muon Beyond Normalized Momentumcited by 0