mlsanityData to 7 Oct 2026

EMO: Frustratingly Easy Progressive Training of Extendable MoE

arXiv 2605.13247 ↗First posted 13 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 43 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

37 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core8 Mar
  2. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  3. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping30 Apr
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  7. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 30

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2024Mixtral of Expertscited by 1,785
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
18 more
  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  9. 2021Hash Layers For Large Sparse Modelscited by 236
  10. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  11. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  12. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  13. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  14. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  15. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  16. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  17. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  18. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
  2. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  3. 2024Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
  4. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  5. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  7. 2026Hyperparameter Scaling Laws Across MoE Sparsity
  8. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource