mlsanityData to 14 Oct 2026

The Big Send-off: Scalable and Performant Collectives for Deep Learning

arXiv 2504.18658 ↗First posted 25 Apr 2025cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Collective communication is becoming increasingly important in data center and supercomputer workloads with an increase in distributed AI related jobs. However, existing libraries that provide collective support such as NCCL, RCCL, and Cray-MPICH exhibit several performance and scalability limitations on modern GPU supercomputers. To address these challenges, we introduce the Performant Collective Communication Library (PCCL), specifically targeted for distributed deep learning (DL) workloads. PCCL provides highly optimized implementations of key collectives used in distributed DL: all-gather, reduce-scatter, and all-reduce. PCCL uses a hierarchical design with learning-based adaptive selection of the best performing algorithms to scale efficiently to thousands of GPUs. It achieves substantial performance speedups over RCCL on 2048 GCDs of Frontier -- up to 168x for reduce-scatter, 33x for all-gather and 10x for all-reduce. More modest but still significant gains up to 5.7x over NCCL are observed on Perlmutter. These gains translate directly to performance improvement of production DL workloads: up to 4.9x speedup over RCCL in DeepSpeed ZeRO-3 training, and up to 2.4x speedup in DDP training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 40 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

17 of 40 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  8. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  9. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  10. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  11. 2020Synthesizing Optimal Collective Algorithmscited by 30
  12. 2021AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learningcited by 15
5 more
  1. 2025Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputerscited by 7
  2. 2024Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnectscited by 6
  3. 2023Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Trainingcited by 5
  4. 2024Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compressioncited by 5
  5. 2024HiCCL: A Hierarchical Collective Communication Librarycited by 3

Built on by

Not cited yet.