# Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects

arXiv: https://arxiv.org/abs/2408.14090 · HTML: https://mlsanity.com/paper/2408.14090/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 53 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Builds on (most cited first, 8 of 8)

- [Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect](https://mlsanity.com/paper/1903.04611/) (2019)
- [Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs](https://mlsanity.com/paper/1711.05979/) (2017)
- [An In-Depth Analysis of the Slingshot Interconnect](https://mlsanity.com/paper/2008.08886/) (2020)
- [LEONARDO: A Pan-European Pre-Exascale Supercomputer for HPC and AI Applications](https://mlsanity.com/paper/2307.16885/) (2023)
- [Mitigating Network Noise on Dragonfly Networks through Application-Aware Routing](https://mlsanity.com/paper/1909.07865/) (2019)
- [Noise in the Clouds: Influence of Network Performance Variability on Application Scalability](https://mlsanity.com/paper/2210.15315/) (2022)
- [Multi-GPU aggregation-based AMG preconditioner for iterative linear solvers](https://mlsanity.com/paper/2303.02352/) (2023)
- [Interconnect Bandwidth Heterogeneity on AMD MI250x and Infinity Fabric](https://mlsanity.com/paper/2302.14827/) (2023)

## Built on by (most cited first, 6 of 6)

- [SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks](https://mlsanity.com/paper/2605.00515/) (2026)
- [Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks](https://mlsanity.com/paper/2603.05692/) (2026)
- [GPU-centric Communication Schemes for HPC and ML Applications](https://mlsanity.com/paper/2503.24230/) (2025)
- [The Big Send-off: Scalable and Performant Collectives for Deep Learning](https://mlsanity.com/paper/2504.18658/) (2025)
- [Accelerating Sharded Data Parallelism at Scale with Federated Learning](https://mlsanity.com/paper/2609.20359/) (2026)
- [Conduit: An Experience Data Plane for Distributed Reinforcement Learning](https://mlsanity.com/paper/2609.24456/) (2026)

## Abstract (arXiv)

Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networks, with bandwidths up to a few terabits per second. However, gauging performance expectations and maximizing system efficiency is challenging due to different technologies, design options, and software layers. This paper comprehensively characterizes three supercomputers - Alps, Leonardo, and LUMI - each with a unique architecture and design. We focus on performance evaluation of intra-node and inter-node interconnects on up to 4096 GPUs, using a mix of intra-node and inter-node benchmarks. By analyzing its limitations and opportunities, we aim to offer practical guidance to researchers, system architects, and software developers dealing with multi-GPU supercomputing. Our results show that there is untapped bandwidth, and there are still many opportunities for optimization, ranging from network to software optimization.