# GC3: An Optimizing Compiler for GPU Collective Communication

arXiv: https://arxiv.org/abs/2201.11840 · HTML: https://mlsanity.com/paper/2201.11840/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2022.

## Built on by (most cited first, 7 of 7)

- [On Optimizing the Communication of Model Parallelism](https://mlsanity.com/paper/2211.05322/) (2022)
- [Impact of RoCE Congestion Control Policies on Distributed Training of DNNs](https://mlsanity.com/paper/2207.10898/) (2022)
- [Efficient Direct-Connect Topologies for Collective Communications](https://mlsanity.com/paper/2202.03356/) (2022)
- [UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training](https://mlsanity.com/paper/2604.19241/) (2026)
- [TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning](https://mlsanity.com/paper/2304.05301/) (2023)
- [Don't Let a Few Network Failures Slow the Entire AllReduce](https://mlsanity.com/paper/2606.01680/) (2026)
- [T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator](https://mlsanity.com/paper/2610.07098/) (2026)

## Abstract (arXiv)

Machine learning models made up of millions or billions of parameters are trained and served on large multi-GPU systems. As models grow in size and execute on more GPUs, the collective communications used in these applications become a bottleneck. Custom collective algorithms optimized for both particular network topologies and application specific communication patterns can alleviate this bottleneck and help these applications scale. However, correctly and efficiently implementing custom algorithms is challenging. This paper introduces GC3, a system for programmable GPU communication. GC3 provides a domain specific language for writing collective communication algorithms and an optimizing compiler for lowering them to an executable form, which can be executed efficiently and flexibly in an interpreter based runtime. We used GC3 to write novel collective algorithms for AllReduce and AllToAll that are up to 1.9× and 1.3× faster than hand-optimized implementations, respectively.