mlsanityData to 14 Oct 2026

ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training

arXiv 2610.04140 ↗First posted 2 Oct 2026cs.LG cs.CL math.OCcited by 0

On the map: Mixture-of-experts → MoE large language models also fits: MoE theory and gating functions

Abstract (arXiv)

Mixture-of-experts (MoE) language models send each token to a few experts, so each expert is trained on a different part of the data, and this part changes during training. With a shared learning rate, Muon applies updates of roughly the same size to expert matrices of the same shape, even when an expert's update is poorly aligned with its current gradient. We here propose ExpertMuon-Compass (Compass), which multiplies the Muon step of each expert by two factors. A family factor compares the cosine between the expert's orthogonalized update and its gradient with the same cosine for the other experts in its layer. A scalar radius aggregates the alignment between corresponding rows of the update and gradient into one step-length multiplier. Compass keeps the update direction and the momentum buffer of Muon. In pretraining on FineWeb-Edu, Compass with Nesterov momentum on all matrices performs as well as or better than Muon, NorMuon, and other optimizers, with weight decay matched to NorMuon in the longer runs. Adding its factors to NorMuon gives the same or a lower loss than NorMuon. Compass is the most effective when the data seen by each expert varies during training, for example, as when the languages of a multilingual corpus arrive in separate blocks. With Compass, the expert load stays balanced, and the router assigns tokens to experts more decisively. We also prove a perturbation bound for the two factors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 13 of its 21 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

13 of 21 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  2. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  3. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer15 Oct
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 9

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  3. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  4. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  5. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  6. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  7. 2026Convergence of Muon with Newton-Schulzcited by 36
  8. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  9. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling
  2. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  3. 2025μ-Parametrization for Mixture of Experts
  4. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
  5. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  6. 2025Advancing Expert Specialization for Better MoE
  7. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  8. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource