mlsanityData to 7 Oct 2026

Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism

arXiv 2602.04870 ↗First posted 4 Feb 2026cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large language models have transformed many applications but remain expensive to train. Sparse Mixture of Experts (MoE) addresses this through conditional computation, with Expert Parallel (EP) as the standard distributed training method. However, EP has three limitations: communication cost grows linearly with the number of activated experts k, load imbalance affects latency and memory usage, and data-dependent communication requires metadata exchange. We propose Multi-Head LatentMoE and Head Parallel (HP), a new architecture and parallelism achieving O(1) communication cost regardless of k, completely balanced traffic, and deterministic communication, all while remaining compatible with EP. To accelerate Multi-Head LatentMoE, we propose IO-aware routing and expert computation. Compared to MoE with EP, Multi-Head LatentMoE with HP trains up to 1.61× faster while having identical performance. With doubled granularity, it achieves higher overall performance while still being 1.11× faster. Our method makes multi-billion-parameter foundation model research more accessible.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 23 of its 35 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

23 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts26 Jan
  2. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models21 Jan
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2024Mixtral of Expertscited by 1,785
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
8 more
  1. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  2. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  3. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  4. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  5. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  6. 2024Multi-Head Mixture-of-Expertscited by 30
  7. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  8. 2024MH-MoE: Multi-Head Mixture-of-Expertscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelismcited by 0
  2. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configurationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  2. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
  3. 2022SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
  4. 2026Memory-Efficient Expert Routing for Distributed MoE Training
  5. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
  6. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training
  7. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
  8. 2024LocMoE: A Low-Overhead MoE for Large Language Model Training