mlsanityData to 14 Oct 2026

Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

arXiv 2605.06206 ↗First posted 7 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing end-to-end prefill latency by up to 5.88x, TTFT by 3.66x, and TPOT by 1.53x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 25 of its 34 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

25 of 34 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts26 Jan
  2. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference29 Sep
  3. 2025FlashMoE: Fast Distributed MoE in a Single Kernel5 Jun
  4. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling6 Mar
  5. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts27 Feb
  6. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  7. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  8. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  9. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  12. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
3 more
  1. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  2. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  3. 2025BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inferencecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
  2. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  3. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Models
  4. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  6. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
  7. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
  8. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts