# Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts

arXiv: https://arxiv.org/abs/2502.19811 · HTML: https://mlsanity.com/paper/2502.19811/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [FastMoE: A Fast Mixture-of-Expert Training System](https://mlsanity.com/paper/2103.13262/) (2021)
- [Taming Sparsely Activated Transformer with Stochastic Experts](https://mlsanity.com/paper/2110.04260/) (2021)
- [The Cost of Training NLP Models: A Concise Overview](https://mlsanity.com/paper/2004.08900/) (2020)
- [FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion](https://mlsanity.com/paper/2406.06858/) (2024)
- [Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion](https://mlsanity.com/paper/2302.01392/) (2023)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)
- [HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System](https://mlsanity.com/paper/2203.14685/) (2022)
- [Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers](https://mlsanity.com/paper/2205.14336/) (2022)
- [MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms](https://mlsanity.com/paper/2209.06800/) (2022)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)

## Built on by (most cited first, 30 of 50)

- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production](https://mlsanity.com/paper/2505.11432/) (2025)
- [VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo](https://mlsanity.com/paper/2508.02317/) (2025)
- [LLM-42: Enabling Determinism in LLM Inference with Verified Speculation](https://mlsanity.com/paper/2601.17768/) (2026)
- [FlashMoE: Fast Distributed MoE in a Single Kernel](https://mlsanity.com/paper/2506.04667/) (2025)
- [Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering](https://mlsanity.com/paper/2504.19519/) (2025)
- [HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs](https://mlsanity.com/paper/2504.03871/) (2025)
- [Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference](https://mlsanity.com/paper/2510.05497/) (2025)
- [LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training](https://mlsanity.com/paper/2602.11686/) (2026)
- [Grouter: Decoupling Routing from Representation for Accelerated MoE Training](https://mlsanity.com/paper/2603.06626/) (2026)
- [UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training](https://mlsanity.com/paper/2604.19241/) (2026)
- [Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts](https://mlsanity.com/paper/2507.15465/) (2025)
- [Understanding and Improving Communication Performance in Multi-node LLM Inference](https://mlsanity.com/paper/2511.09557/) (2025)
- [Hierarchical Mixture-of-Experts with Two-Stage Optimization](https://mlsanity.com/paper/2605.08292/) (2026)
- [ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning](https://mlsanity.com/paper/2605.08639/) (2026)
- [FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training](https://mlsanity.com/paper/2510.00207/) (2025)
- [FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concern](https://mlsanity.com/paper/2510.15882/) (2025)
- [SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs](https://mlsanity.com/paper/2601.20273/) (2026)
- [Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training](https://mlsanity.com/paper/2606.11867/) (2026)
- [HipKittens: Fast and Furious AMD Kernels](https://mlsanity.com/paper/2511.08083/) (2025)
- [Fine-grained MoE Load Balancing with Linear Programming](https://mlsanity.com/paper/2511.16947/) (2025)
- [Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap](https://mlsanity.com/paper/2512.10236/) (2025)
- [MoEless: Efficient MoE LLM Serving via Serverless Computing](https://mlsanity.com/paper/2603.06350/) (2026)
- [ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving](https://mlsanity.com/paper/2606.00735/) (2026)
- [Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Serving](https://mlsanity.com/paper/2607.18631/) (2026)
- [Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts](https://mlsanity.com/paper/2607.19539/) (2026)
- [BootSeer: Analyzing and Mitigating Initialization Bottlenecks in Large-Scale LLM Training](https://mlsanity.com/paper/2507.12619/) (2025)
- [NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining](https://mlsanity.com/paper/2604.06956/) (2026)
- [Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants](https://mlsanity.com/paper/2604.18616/) (2026)
- [DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training](https://mlsanity.com/paper/2605.01989/) (2026)

## Abstract (arXiv)

Mixture-of-experts (MoE) has been extensively employed to scale large language models to trillion-plus parameters while maintaining a fixed computational cost. The development of large MoE models in the distributed scenario encounters the problem of large communication overhead. The inter-device communication of a MoE layer can occupy 47% time of the entire model execution with popular models and frameworks. Therefore, existing methods suggest the communication in a MoE layer to be pipelined with the computation for overlapping. However, these coarse grained overlapping schemes introduce a notable impairment of computational efficiency and the latency concealing is sub-optimal. To this end, we present COMET, an optimized MoE system with fine-grained communication-computation overlapping. Leveraging data dependency analysis and task rescheduling, COMET achieves precise fine-grained overlapping of communication and computation. Through adaptive workload assignment, COMET effectively eliminates fine-grained communication bottlenecks and enhances its adaptability across various scenarios. Our evaluation shows that COMET accelerates the execution of a single MoE layer by 1.96× and for end-to-end execution, COMET delivers a 1.71× speedup on average. COMET has been adopted in the production environment of clusters with ten-thousand-scale of GPUs, achieving savings of millions of GPU hours.