# GPU-centric Communication Schemes for HPC and ML Applications

arXiv: https://arxiv.org/abs/2503.24230 · HTML: https://mlsanity.com/paper/2503.24230/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 95 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Deep Learning of Representations: Looking Forward](https://mlsanity.com/paper/1305.0445/) (2013)
- [Scaling Distributed Machine Learning with In-Network Aggregation](https://mlsanity.com/paper/1903.06701/) (2019)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [CosmoFlow: Using Deep Learning to Learn the Universe at Scale](https://mlsanity.com/paper/1808.04728/) (2018)
- [An In-Depth Analysis of the Slingshot Interconnect](https://mlsanity.com/paper/2008.08886/) (2020)
- [Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchip](https://mlsanity.com/paper/2408.11556/) (2024)
- [A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks](https://mlsanity.com/paper/2111.04949/) (2021)
- [Demystifying the Communication Characteristics for Distributed Transformer Models](https://mlsanity.com/paper/2408.10197/) (2024)
- [Distributed Matrix-Based Sampling for Graph Neural Network Training](https://mlsanity.com/paper/2311.02909/) (2023)
- [Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects](https://mlsanity.com/paper/2408.14090/) (2024)
- [MPIX Stream: An Explicit Solution to Hybrid MPI+X Programming](https://mlsanity.com/paper/2208.13707/) (2022)
- [The Landscape of GPU-Centric Communication](https://mlsanity.com/paper/2409.09874/) (2024)
- [Intel(R) SHMEM: GPU-initiated OpenSHMEM using SYCL](https://mlsanity.com/paper/2409.20476/) (2024)
- [Understanding Data Movement in AMD Multi-GPU Systems with Infinity Fabric](https://mlsanity.com/paper/2410.00801/) (2024)
- [Extreme Scale De Novo Metagenome Assembly](https://mlsanity.com/paper/1809.07014/) (2018)
- [HipBone: A performance-portable GPU-accelerated C++ version of the NekBone benchmark](https://mlsanity.com/paper/2202.12477/) (2022)
- [Exploring GPU Stream-Aware Message Passing using Triggered Operations](https://mlsanity.com/paper/2208.04817/) (2022)
- [Exploring Fully Offloaded GPU Stream-Aware Message Passing](https://mlsanity.com/paper/2306.15773/) (2023)
- [Understanding GPU Triggering APIs for MPI+X Communication](https://mlsanity.com/paper/2406.05594/) (2024)

## Abstract (arXiv)

Compute nodes on modern heterogeneous supercomputing systems comprise CPUs, GPUs, and high-speed network interconnects (NICs). Parallelization is identified as a technique for effectively utilizing these systems to execute scalable simulation and deep learning workloads. The resulting inter-process communication from the distributed execution of these parallel workloads is one of the key factors contributing to its performance bottleneck. Most programming models and runtime systems enabling the communication requirements on these systems support GPU-aware communication schemes that move the GPU-attached communication buffers in the application directly from the GPU to the NIC without staging through the host memory. A CPU thread is required to orchestrate the communication operations even with support for such GPU-awareness. This survey discusses various available GPU-centric communication schemes that move the control path of the communication operations from the CPU to the GPU. This work presents the need for the new communication schemes, various GPU and NIC capabilities required to implement the schemes, and the potential use-cases addressed. Based on these discussions, challenges involved in supporting the exhibited GPU-centric communication schemes are discussed.