mlsanityData to 14 Oct 2026

Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

arXiv 2605.00254 ↗First posted 30 Apr 2026cs.NI cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-experts (MoE) architectures have turned LLM serving into a cluster-scale workload in which communication consumes a considerable portion of LLM serving runtime. This has prompted industry to invest heavily in expensive high-bandwidth scale-up networks. We question whether such costly infrastructure is strictly necessary. We present the first systematic cross-layer analysis of network cost-effectiveness for MoE LLM serving, comparing four representative XPU (e.g., GPU/TPU) topologies (scale-up, scale-out, 3D torus, and 3D full-mesh). We find that lower-cost switchless topologies are more cost-effective than the scale-up topology across all serving scenarios explored, improving cost-effectiveness by 20.6-56.2%. In particular, the 3D full-mesh topology is Pareto-optimal in terms of the performance-cost tradeoff. We also find that current scale-up link bandwidths are over-provisioned: reducing the link bandwidth improves throughput per cost by up to 27%. A forward-looking analysis of upcoming GPU generations indicates that the cost-performance advantage of switchless networks will likely persist.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 31 of its 68 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

31 of 68 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference15 Dec
  2. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts21 Jul
  3. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts17 Jun
  4. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training7 Jan
  5. 2023Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters22 Jul

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  6. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2023Speculative Decoding with Big Little Decodercited by 164
  11. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  12. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
14 more
  1. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  2. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 31
  3. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  4. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scalingcited by 19
  5. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 16
  6. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  7. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  8. 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 14
  9. 2023SuperServe: Fine-Grained Inference Serving for Unpredictable Workloadscited by 12
  10. 2023Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problemcited by 11
  11. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  12. 2025UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecturecited by 6
  13. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  14. 2023Efficient All-to-All Collective Communication Schedules for Direct-Connect Topologiescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
  2. 2026RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving
  3. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
  4. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  5. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  6. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems
  7. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  8. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration