mlsanityData to 14 Oct 2026

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

arXiv 1903.04611 ↗First posted 11 Mar 2019cs.AR cs.DC cs.NIcited by 41

Not on the map. Why?

Abstract (arXiv)

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 6; 2023: 5; 2024: 11; 2025: 8; 2026: 5.

41 papers.

Sort:

By area:

  1. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  2. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 48
  3. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  4. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 16
  5. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  6. 2021PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Managementcited by 15
  7. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  8. 2023Aligning Language Models with Offline Learning from Human Feedbackcited by 14
  9. 2024Federated Learning Priorities Under the European Union Artificial Intelligence Actcited by 13
  10. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  11. 2023G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrationscited by 10
  12. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
29 more
  1. 2023Zero redundancy distributed learning with differential privacycited by 7
  2. 2022Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Serverscited by 6
  3. 2024Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnectscited by 6
  4. 2022Efficient Direct-Connect Topologies for Collective Communicationscited by 5
  5. 2024Deep Learning Library Testing: Definition, Methods and Challengescited by 5
  6. 2024Retrofitting Temporal Graph Neural Networks with Transformercited by 5
  7. 2021Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)cited by 4
  8. 2021MG-GCN: Scalable Multi-GPU GCN Training Frameworkcited by 4
  9. 2021CGX: Adaptive System Support for Communication-Efficient Deep Learningcited by 4
  10. 2023Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environmentcited by 4
  11. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughputcited by 4
  12. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decodingcited by 4
  13. 2021An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networkscited by 2
  14. 2023GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusterscited by 2
  15. 2024GPU Cluster Scheduling for Network-Sensitive Deep Learningcited by 2
  16. 2025EQuARX: Efficient Quantized AllReduce in XLA for Distributed Machine Learning Accelerationcited by 2
  17. 2025WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Librarycited by 2
  18. 2022RAMP: A Flat Nanosecond Optical Network and MPI Operations for Distributed Deep Learning Systemscited by 1
  19. 2024Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platformscited by 1
  20. 2024FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Modelscited by 1
  21. 2025Intra-DP: A High Performance Collaborative Inference System for Mobile Edge Computingcited by 1
  22. 2025G-Core: A Simple, Scalable and Balanced RLHF Trainercited by 1
  23. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottleneckscited by 1
  24. 2025Accelerating MoE Model Inference with Expert Shardingcited by 0
  25. 2025Hardware-Aware Data and Instruction Mapping for AI Tasks: Balancing Parallelism, I/O and Memory Tradeoffscited by 0
  26. 2026COMPASS-ABS: Reducing Fragmentation in Shared GPU Clusters for Deep Learning Training Workloadscited by 0
  27. 2026The Life of a Token: from Words to Bits on the Wirecited by 0
  28. 2026Conduit: An Experience Data Plane for Distributed Reinforcement Learningcited by 0
  29. 2026EMA: Elastic and Performance Transparent Memory Across GPUscited by 0