# MSCCL++: Rethinking GPU Communication Abstractions for AI Inference

arXiv: https://arxiv.org/abs/2504.09014 · HTML: https://mlsanity.com/paper/2504.09014/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 9 of 9)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [Synthesizing Optimal Collective Algorithms](https://mlsanity.com/paper/2008.08708/) (2020)
- [BLITZSCALE: Fast and Live Large Model Autoscaling with O(1) Host Caching](https://mlsanity.com/paper/2412.17246/) (2024)
- [Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem](https://mlsanity.com/paper/2305.13479/) (2023)

## Built on by (most cited first, 10 of 10)

- [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](https://mlsanity.com/paper/2601.17654/) (2026)
- [GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving](https://mlsanity.com/paper/2606.13501/) (2026)
- [NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding](https://mlsanity.com/paper/2605.21100/) (2026)
- [UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods](https://mlsanity.com/paper/2607.06202/) (2026)
- [Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration](https://mlsanity.com/paper/2608.24664/) (2026)
- [mKernel: Fast Multi-GPU, Multi-Node Fused Kernels](https://mlsanity.com/paper/2609.13585/) (2026)
- [The Life of a Token: from Words to Bits on the Wire](https://mlsanity.com/paper/2609.19924/) (2026)
- [Purlin: Separating Orchestration from the Datapath of Collectives](https://mlsanity.com/paper/2609.36954/) (2026)
- [Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs](https://mlsanity.com/paper/2609.40093/) (2026)
- [T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator](https://mlsanity.com/paper/2610.07098/) (2026)

## Abstract (arXiv)

AI applications increasingly run on fast-evolving, heterogeneous hardware to maximize performance, but general-purpose libraries lag in supporting these features. Performance-minded programmers often build custom communication stacks that are fast but error-prone and non-portable. This paper introduces MSCCL++, a design methodology for developing high-performance, portable communication kernels. It provides (1) a low-level, performance-preserving primitive interface that exposes minimal hardware abstractions while hiding the complexities of synchronization and consistency, (2) a higher-level DSL for application developers to implement workload-specific communication algorithms, and (3) a library of efficient algorithms implementing the standard collective API, enabling adoption by users with minimal expertise. Compared to state-of-the-art baselines, MSCCL++ achieves geomean speedups of 1.7× (up to 5.4×) for collective communication and 1.2× (up to 1.38×) for AI inference workloads. MSCCL++ is in production of multiple AI services provided by Microsoft Azure, and has also been adopted by RCCL, the GPU collective communication library maintained by AMD. MSCCL++ is open source and available at https://github.com/microsoft/mscclpp . Our two years of experience with MSCCL++ suggests that its abstractions are robust, enabling support for new hardware features, such as multimem, within weeks of development.