mlsanityData to 14 Oct 2026

NCCLZ: Compression-Enabled GPU Collectives with Decoupled Quantization and Entropy Coding

arXiv 2605.12396 ↗First posted 12 May 2026cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Collective communication is a major bottleneck for multi-node GPU workloads in scientific computing and distributed deep learning, especially when inter-node bandwidth is limited. Although NCCL provides optimized GPU-centric collectives, large messages can still dominate end-to-end performance. Existing compression-enabled collective libraries either rely on MPI-based stacks that cannot fully exploit NCCL, omit entropy coding, or tightly couple full compressors with communication primitives, limiting compression ratio, flexibility, and communication-computation overlap. This paper presents NCCLZ, a compression-enabled GPU collectives that decouples quantization and entropy coding and integrates them at different layers of the stack. NCCLZ places quantization at the interface, embeds entropy coding into NCCL primitives, uses a lightweight device-side selector to choose coding strategies, and overlaps compression with communication to reduce exposed overhead. Experiments on scientific datasets, training gradients, and synthetic workloads show up to 9.65x speedup over NCCL and up to 3.34x improvement over prior compression-assisted collective libraries.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 42 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

13 of 42 references.

References · 13

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  5. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 57
  6. 2020Is Network the Bottleneck of Distributed Training?cited by 20
  7. 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 20
  8. 2020cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Datacited by 18
  9. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 11
  10. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
  11. 2023gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusterscited by 6
  12. 2024Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compressioncited by 5
1 more
  1. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UCCL-Zip: Lossless Compression Supercharged GPU Communication
  2. 2023FZ-GPU: A Fast and High-Ratio Lossy Compressor for Scientific Computing Applications on GPUs
  3. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
  4. 2026ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
  5. 2026Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
  6. 2023Evaluation and Optimization of Gradient Compression for Distributed Deep Learning
  7. 2026EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates
  8. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operations