HetCCL: Accelerating LLM Training with Heterogeneous GPUs
Not on the map. Why?
Abstract (arXiv)
The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 61 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
30 of 61 references.
References · 30
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
18 more
- 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
- 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
- 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
- 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
- 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
- 2024Characterization of Large Language Model Development in the Datacentercited by 46
- 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
- 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 16
- 2022AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awarenesscited by 13
- 2025An Extensible Software Transport Layer for GPU Networkingcited by 10
- 2025Collective Communication for 100k+ GPUscited by 4
- 2020HetSeq: Distributed GPU Training on Heterogeneous Infrastructurecited by 3
- 2024HiCCL: A Hierarchical Collective Communication Librarycited by 3
- 2024Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Modelscited by 3
- 2025H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chipscited by 2
- 2025KAITIAN: A Unified Communication Framework for Enabling Efficient Collaboration Across Heterogeneous Accelerators in Embodied AI Systemscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.