mlsanityData to 14 Oct 2026

GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference

arXiv 2509.25041 ↗First posted 29 Sep 2025cs.DCcited by 13

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparse Mixture of Experts (SMoE) enables scalable parameter growth in large language models (LLMs) by selectively activating a subset of experts, and its large parameter count necessitates distributed deployment for inference. However, distributed inference faces a critical dilemma: although communication overhead constitutes the primary bottleneck, reducing it often exacerbates computational load imbalance, leading to resource waste. In this paper, we present GRACE-MoE, which stands for Grouping and Replication with Locality-Aware Routing for SMoE inference. GRACE-MoE is a lossless co-optimization framework that integrates expert grouping to reduce communication and dynamic replication to correct load skew, together with locality-aware routing to resolve replica selection. To underpin this coordinated optimization in multi-node settings, GRACE-MoE adopts a hierarchical sparse communication design that reduces cross-node traffic while implicitly aligning execution across nodes, thereby mitigating synchronization overhead. Experiments on diverse models and multi-node, multi-GPU environments demonstrate that GRACE-MoE efficiently reduces end-to-end inference latency, achieving up to 4.66x speedup over existing systems, and the code will be released upon acceptance.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 33 of its 46 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

33 of 46 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design2 Apr
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  4. 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models5 Jul
  5. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  6. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  9. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  10. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  11. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May

Other references · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
10 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  6. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  7. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  8. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  9. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  10. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2026PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetchingcited by 4
  2. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decodingcited by 4
  3. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 3
  4. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  5. 2025Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networkscited by 1
  6. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inferencecited by 1
  7. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  8. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Modelscited by 0
  9. 2026Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inferencecited by 0
  10. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  11. 2026UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpodscited by 0
  12. 2026HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inferencecited by 0
1 more
  1. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shufflingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  2. 2026FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
  3. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  4. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  5. 2026OrderMoE: An expert similarity driven distributed edge MoE inference
  6. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
  7. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  8. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefill