mlsanityData to 14 Oct 2026

Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

arXiv 2607.05116 ↗First posted 6 Jul 2026cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

As MoE models scale to hundreds of experts, placement and pruning decisions increasingly dictate communication volume, affecting the performance of distributed inference across GPUs and nodes. We propose CAP (Communication-Aware Assignment and Pruning), a framework that considers computation, communication and accuracy together for efficient MoE inference through expert placement and pruning. It consists of three components: (1) Co-activation driven expert placement, which groups frequently co-activated experts to reduce inter-device and inter-node communication; (2) Communicationcomputation trade-off adjustment, which generates placements with different computational load and communication volume; and (3) Communication-aware expert pruning, which selectively removes routing destinations to reduce communication with limited accuracy degradation. By combining these components, CAP selects an efficient operating strategy for different hardware configurations. Across our single-node and multi-node experiments, it achieves 1.23x - 1.86 x throughput improvement over DeepSeek EPLB and sequential placement in vLLM, and preserves better model accuracy at the same target speedup under lossy acceleration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 31 of its 37 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

31 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference29 Sep
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  3. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  5. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  6. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  7. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2024Mixtral of Expertscited by 1,823
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
11 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  2. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  3. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  5. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  6. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  7. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  8. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 37
  9. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  10. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  11. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling
  4. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  5. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  6. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference
  7. 2026Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference
  8. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference