mlsanityData to 14 Oct 2026

TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication

arXiv 2608.27874 ↗First posted 28 Aug 2026cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Hierarchical two-hop dispatch can reduce slow-fabric traffic in expert-parallel Mixture-of-Experts training, but it adds a second collective and an arrival-side operator chain. We present a cost model for screening that trade at the communication-call level, bounded by validation gates that withdraw a capability in code when they fail rather than reporting a caveat. At a reference geometry with 16 groups of 8 ranks, q=3, H=2048, and 4096 tokens per rank, the corrected effective breakeven hierarchy ratio is 3.98 for the measured PyTorch arrival chain, 1.49 for a hypothetical fused target, and 1.10 at zero implementation overhead. These are ratio-only sensitivity results, not deployment predictions: platform A measures 1.03, platform B has no separated fast/slow measurement, and neither machine measured here reaches the hierarchical regime. Four communication-level corpora pass their gates; a drift probe and the step-level gate fail. The latter failure is enforced in code, so we make no training-throughput prediction. The enabling routing constraint fixes per-token fan-out and per-selected-group quota, while aggregate per-peer counts remain data-dependent. Its measured validation-loss cost is small but nonzero (+0.0034 nats); downstream equivalence is reported with incomplete estimator provenance and is therefore not independently reconstructible from the artifact. Code, calibration constants and the validation gates are at https://github.com/weich97/TerraceMoE-simulator.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 20 of its 44 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

20 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms18 Aug
  2. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap13 Aug
  3. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production16 May
  4. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures14 May
  5. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  6. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  7. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 12

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  7. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  8. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  9. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 18
  10. 2022dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Trainingcited by 12
  11. 2023Proteus: Simulating the Performance of Distributed DNN Trainingcited by 10
  12. 2019Node-Aware Improvements to Allreducecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
  2. 2026Memory-Efficient Expert Routing for Distributed MoE Training
  3. 2025RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
  4. 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
  5. 2026RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
  6. 2026Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend
  7. 2025FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models
  8. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization