# Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

arXiv: https://arxiv.org/abs/2605.00254 · HTML: https://mlsanity.com/paper/2605.00254/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 31 of its 68 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale](https://mlsanity.com/paper/2408.05499/) (2024)
- [TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs](https://mlsanity.com/paper/2202.00433/) (2022)
- [Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling](https://mlsanity.com/paper/2501.08219/) (2025)
- [Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models](https://mlsanity.com/paper/2110.04478/) (2021)
- [APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving](https://mlsanity.com/paper/2411.17651/) (2024)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference](https://mlsanity.com/paper/2505.11329/) (2025)
- [Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters](https://mlsanity.com/paper/2307.12169/) (2023)
- [Utility-Driven Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2506.20675/) (2025)
- [SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads](https://mlsanity.com/paper/2312.16733/) (2023)
- [Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem](https://mlsanity.com/paper/2305.13479/) (2023)
- [Swing: Short-cutting Rings for Higher Bandwidth Allreduce](https://mlsanity.com/paper/2401.09356/) (2024)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training](https://mlsanity.com/paper/2501.03905/) (2025)
- [UB-Mesh: a Hierarchically Localized nD-FullMesh Datacenter Network Architecture](https://mlsanity.com/paper/2503.20377/) (2025)
- [TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning](https://mlsanity.com/paper/2304.05301/) (2023)
- [Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts](https://mlsanity.com/paper/2507.15465/) (2025)

## Abstract (arXiv)

Mixture-of-experts (MoE) architectures have turned LLM serving into a cluster-scale workload in which communication consumes a considerable portion of LLM serving runtime. This has prompted industry to invest heavily in expensive high-bandwidth scale-up networks. We question whether such costly infrastructure is strictly necessary. We present the first systematic cross-layer analysis of network cost-effectiveness for MoE LLM serving, comparing four representative XPU (e.g., GPU/TPU) topologies (scale-up, scale-out, 3D torus, and 3D full-mesh). We find that lower-cost switchless topologies are more cost-effective than the scale-up topology across all serving scenarios explored, improving cost-effectiveness by 20.6-56.2%. In particular, the 3D full-mesh topology is Pareto-optimal in terms of the performance-cost tradeoff. We also find that current scale-up link bandwidths are over-provisioned: reducing the link bandwidth improves throughput per cost by up to 27%. A forward-looking analysis of upcoming GPU generations indicates that the cost-performance advantage of switchless networks will likely persist.