mlsanityData to 14 Oct 2026

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

arXiv 2606.19025 ↗First posted 17 Jun 2026cs.LG cs.AI cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators. Mixture-of-Experts (MoEs) architectures partially decouple model capacity from per-token compute. This efficiency alone does not make MoE training feasible over ordinary Internet links or loosely connected commodity hardware since active expert routing still assumes high-speed datacenter fabrics. Low-communication methods such as DiLoCo and Photon reduce synchronization frequency across distributed sites, mitigating bandwidth constraints, yet still require full model replicas at every site. This creates a mismatch: modern MoEs have sparse data paths, but their distributed training infrastructure remains communication-dense and memory-inefficient, limiting attempts to pool geographically distributed compute. In this work, we introduce FoMoE, a system that breaks the full-replica paradigm by partitioning expert layers across workers and skipping non-resident experts during local training. We demonstrate that FoMoE: (I) reduces communication costs by up to 1.42x over efficient baselines and 45.44x over Distributed Data Parallelism (DDP) via partial expert replication in controlled regimes; (II) achieves empirical throughput speedups of up to 1.4x through the skip-token mechanism; and (III) shows stable routing in the trained regimes and projects the communication/memory benefits to 100B-scale configurations through system modeling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 50 of its 70 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

50 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  2. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  3. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts10 Feb

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2019Advances and Open Problems in Federated Learningcited by 3,046
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  12. 2016Deep Learning with Differential Privacycited by 2,728
35 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  13. 2021Hash Layers For Large Sparse Modelscited by 237
  14. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  15. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  16. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  17. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  18. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  19. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 34
  20. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 28
  21. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  22. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  23. 2024OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Trainingcited by 24
  24. 2020Specialized federated learning using a mixture of expertscited by 23
  25. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Expertscited by 21
  26. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretrainingcited by 12
  27. 2024DiPaCo: Distributed Path Compositioncited by 10
  28. 2024Convergence of Distributed Adaptive Optimization with Local Updatescited by 10
  29. 2025dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysiscited by 9
  30. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 9
  31. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 5
  32. 2025DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Modelscited by 4
  33. 2025DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Clustercited by 4
  34. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  35. 2026Unifying Local Communications and Local Updates for LLM Pretrainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2024LocMoE: A Low-Overhead MoE for Large Language Model Training
  5. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  6. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  7. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  8. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving