Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback
Not on the map. Why?
Abstract (arXiv)
SignMuon compresses the Muon update to one bit per parameter by taking its elementwise sign, providing the most direct way to run a matrix-aware optimizer under an extremely low communication budget. It outperforms SignSGD in practice, yet it can ascend even on a linear function. Signing the gradient before the Linear Minimization Oracle (LMO), rather than after, does not repair this: we construct a small explicit instance on which sign-before (MuonUSign) and sign-on-both-sides (MuonSign) ascend as well, so no placement of the sign around the oracle descends in general. Error feedback, the standard remedy for a biased compressor, does not rescue SignMuon: when applied to Muon's output, error feedback can fail for every smoothness constant, step size, and momentum. Applied to the gradient, error feedback does work, and EF21-MuonUSign and EF21-MuonSign attain the standard O(T-1/2) rate for the squared gradient norm on smooth nonconvex problems, the latter at one bit in each direction. Experiments then reverse the ordering: across centralized CIFAR-10, federated CIFAR-10, and the nanoGPT speedrun, the strongest compressed method is consistently sign-after-the-LMO, precisely the placement we prove divergent, with the provably convergent variants trailing it. Compressing after the LMO, a heuristic, matters more at these scales than the guarantee does.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
27 of 39 references.
References · 27
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
- 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
- 2020On Biased Compression for Distributed Learningcited by 140
- 2020Momentum Improves Normalized SGDcited by 136
- 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
- 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
- 2023A Spectral Condition for Feature Learningcited by 101
- 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
- 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
- 2025A Note on the Convergence of Muoncited by 78
15 more
- 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
- 2024Scalable Optimization in the Modular Normcited by 64
- 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
- 2020Stochastic-Sign SGD for Federated Learning with Theoretical Guaranteescited by 34
- 2019Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithmscited by 29
- 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
- 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
- 2025Error Feedback for Muon and Friendscited by 11
- 2026Muon Does Not Converge on Convex Lipschitz Functionscited by 9
- 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
- 2025On Provable Benefits of Muon in Federated Learningcited by 7
- 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
- 2026Communication-Efficient Gluon in Federated Learningcited by 2
- 2026SignMuon: Communication-Efficient Distributed Muon Optimizationcited by 2
- 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 2
Built on by
Not cited yet.