mlsanityData to 14 Oct 2026

NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL

arXiv 2603.13606 ↗First posted 13 Mar 2026cs.DC cs.AI cs.ARcited by 5

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 13 of its 24 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

13 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025UCCL-EP: Portable Expert-Parallel Communication22 Dec
  2. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 10

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2024Mixtral of Expertscited by 1,823
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  9. 2025GPU-Initiated Networking for NCCLcited by 5
  10. 2025Collective Communication for 100k+ GPUscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inferencecited by 2
  2. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  3. 2026NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decodingcited by 0
  4. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUscited by 0
  5. 2026Memory-Efficient Expert Routing for Distributed MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
  2. 2026Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend
  3. 2026UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
  4. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  5. 2026Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
  6. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling
  7. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  8. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference