mlsanityData to 14 Oct 2026

MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training

arXiv 2501.03905 ↗First posted 7 Jan 2025cs.NI cs.LGcited by 7

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Expert (MoE) models outperform conventional models by selectively activating different subnets, named experts, on a per-token basis. This gated computation generates dynamic communications that cannot be determined beforehand, challenging the existing GPU interconnects that remain static during the distributed training process. In this paper, we advocate for a first-of-its-kind system, called MixNet, that unlocks topology reconfiguration during distributed MoE training. Towards this vision, we first perform a production measurement study and show that the MoE dynamic communication pattern has strong locality, alleviating the requirement of global reconfiguration. Based on this, we design and implement a regionally reconfigurable high-bandwidth domain on top of existing electrical interconnects using optical circuit switching (OCS), achieving scalability while maintaining rapid adaptability. We have built a fully functional MixNet prototype with commodity hardware and a customized collective communication runtime that trains state-of-the-art MoE models with in-training topology reconfiguration across 32 A100 GPUs. Large-scale packet-level simulations show that MixNet delivers comparable performance as the non-blocking fat-tree fabric while boosting the training cost efficiency (e.g., performance per dollar) of four representative MoE models by 1.2x-1.5x and 1.9x-2.3x at 100 Gbps and 400 Gbps link bandwidths, respectively.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 18 of its 103 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

18 of 103 references.

References · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  7. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  8. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  9. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  10. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  11. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  12. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
6 more
  1. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  2. 2021Design and Operation of Shared Machine Learning Clusters on Campuscited by 12
  3. 2019Expanding across time to deliver bandwidth efficiency and low latencycited by 7
  4. 2021Gemini: Practical Reconfigurable Datacenter Networks with Topology and Traffic Engineeringcited by 2
  5. 2021Optimal Oblivious Reconfigurable Networkscited by 1
  6. 2023Breaking the VLB Barrier for Oblivious Reconfigurable Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Trainingcited by 1
  2. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  3. 2025Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networkscited by 0
  4. 2026Scheduling Parallel Optical Circuit Switches for AI Trainingcited by 0
  5. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
  6. 2026DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Trainingcited by 0
  7. 2026ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
  2. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  3. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems
  4. 2026Scaling Inference Prefill with High-Radix Photonic Interconnects
  5. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
  6. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
  7. 2026Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlap
  8. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts