mlsanityData to 7 Oct 2026

Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts

arXiv 2502.19811 ↗First posted 27 Feb 2025cs.DC cs.AI cs.LGcited by 46

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-experts (MoE) has been extensively employed to scale large language models to trillion-plus parameters while maintaining a fixed computational cost. The development of large MoE models in the distributed scenario encounters the problem of large communication overhead. The inter-device communication of a MoE layer can occupy 47% time of the entire model execution with popular models and frameworks. Therefore, existing methods suggest the communication in a MoE layer to be pipelined with the computation for overlapping. However, these coarse grained overlapping schemes introduce a notable impairment of computational efficiency and the latency concealing is sub-optimal. To this end, we present COMET, an optimized MoE system with fine-grained communication-computation overlapping. Leveraging data dependency analysis and task rescheduling, COMET achieves precise fine-grained overlapping of communication and computation. Through adaptive workload assignment, COMET effectively eliminates fine-grained communication bottlenecks and enhances its adaptability across various scenarios. Our evaluation shows that COMET accelerates the execution of a single MoE layer by 1.96× and for end-to-end execution, COMET delivers a 1.71× speedup on average. COMET has been adopted in the production environment of clusters with ten-thousand-scale of GPUs, achieving savings of millions of GPU hours.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 38 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

22 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  2. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  4. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2024Mixtral of Expertscited by 1,785
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  12. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
6 more
  1. 2023Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusioncited by 54
  2. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  3. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  4. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  5. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  6. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 32.

46 papers.

Sort:

By area:

  1. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 28
  2. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  3. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  4. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 13
  5. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 9
  6. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7
  7. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  8. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 5
  9. 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Trainingcited by 5
  10. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  11. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 4
  12. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Trainingcited by 2
34 more
  1. 2025Understanding and Improving Communication Performance in Multi-node LLM Inferencecited by 2
  2. 2026SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUscited by 2
  3. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 2
  4. 2026ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learningcited by 2
  5. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  6. 2025FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concerncited by 1
  7. 2025HipKittens: Fast and Furious AMD Kernelscited by 1
  8. 2025Fine-grained MoE Load Balancing with Linear Programmingcited by 1
  9. 2025Design Space Exploration of DMA based Finer-Grain Compute Communication Overlapcited by 1
  10. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  11. 2026ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Servingcited by 1
  12. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Expertscited by 1
  13. 2025BootSeer: Analyzing and Mitigating Initialization Bottlenecks in Large-Scale LLM Trainingcited by 0
  14. 2026NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipeliningcited by 0
  15. 2026Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariantscited by 0
  16. 2026DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Trainingcited by 0
  17. 2026Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUscited by 0
  18. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systemscited by 0
  19. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Modelscited by 0
  20. 2026DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelismcited by 0
  21. 2026Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Modelscited by 0
  22. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoEcited by 0
  23. 2026HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUscited by 0
  24. 2026Resource-aware Computation-Communication Overlap for multi-GPU ML Workloadscited by 0
  25. 2026UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpodscited by 0
  26. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 0
  27. 2026X-Stage: Modeling Post-Issue Backpressure in GPU Communication--Computation Fusioncited by 0
  28. 2026CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Trainingcited by 0
  29. 2026Analytical Resource Management for Fine-grained MoE Computation-Communication Overlapcited by 0
  30. 2026mKernel: Fast Multi-GPU, Multi-Node Fused Kernelscited by 0
  31. 2026Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlapcited by 0
  32. 2026TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Trainingcited by 0
  33. 2026Mixture-of-Kittens: MoE Megakernel for NVL72scited by 0
  34. 2026HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
  2. 2026Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  5. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  6. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  7. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
  8. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism