mlsanityData to 14 Oct 2026

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts

arXiv 2404.05019 ↗First posted 7 Apr 2024cs.LG cs.CL cs.DCcited by 32

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Expert parallelism has emerged as a key strategy for distributing the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple devices, enabling the processing of increasingly large-scale models. However, the All-to-All communication inherent to expert parallelism poses a significant bottleneck, limiting the efficiency of MoE models. Although existing optimization methods partially mitigate this issue, they remain constrained by the sequential dependency between communication and computation operations. To address this challenge, we propose ScMoE, a novel shortcut-connected MoE architecture integrated with an overlapping parallelization strategy. ScMoE decouples communication from its conventional sequential ordering, enabling up to 100% overlap with computation. Compared to the prevalent top-2 MoE baseline, ScMoE achieves speedups of 1.49 times in training and 1.82 times in inference. Moreover, our experiments and analyses indicate that ScMoE not only achieves comparable but in some instances surpasses the model quality of existing approaches.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 68 of its 89 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

68 of 89 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  3. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  6. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  8. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  13. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  14. 2024Mixtral of Expertscited by 1,823
  15. 2020Reformer: The Efficient Transformercited by 1,668
  16. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  17. 2020Rethinking Attention with Performerscited by 1,601
  18. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  19. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  20. 2020On Layer Normalization in the Transformer Architecturecited by 922
  21. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  22. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  23. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  24. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  25. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  26. 2018Universal Transformerscited by 647
  27. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  28. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  29. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  30. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  31. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  32. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  33. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  34. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  35. 2021Hash Layers For Large Sparse Modelscited by 237
  36. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  37. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  38. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 10; 2026: 16.

32 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  3. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  4. 2025LongCat-Flash Technical Reportcited by 58
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  6. 2025Advancing Expert Specialization for Better MoEcited by 39
  7. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  8. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  9. 2025LongCat-Flash-Omni Technical Reportcited by 26
  10. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  11. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  12. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
20 more
  1. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  2. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  3. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Trainingcited by 4
  4. 2026LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexingcited by 2
  5. 2025FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communicationcited by 1
  6. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  7. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  8. 2026ReaLB: Real-Time Load Balancing for Multimodal MoE Inferencecited by 1
  9. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inferencecited by 1
  10. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborativecited by 0
  11. 2026A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoEcited by 0
  12. 2026How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Lawscited by 0
  13. 2026MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Trainingcited by 0
  14. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inferencecited by 0
  15. 2026The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelismcited by 0
  16. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configurationcited by 0
  17. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processingcited by 0
  18. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Trainingcited by 0
  19. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUscited by 0
  20. 2026CAST: History-Adaptive Spectral Clipping for LLM Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
  2. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
  5. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
  6. 2023Revisiting Single-gated Mixtures of Experts
  7. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  8. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference