# ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training

arXiv: https://arxiv.org/abs/2604.27844 · HTML: https://mlsanity.com/paper/2604.27844/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [Understanding Top-k Sparsification in Distributed Deep Learning](https://mlsanity.com/paper/1911.08772/) (2019)
- [Blink: Fast and Generic Collectives for Distributed ML](https://mlsanity.com/paper/1910.04940/) (2019)
- [70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)](https://mlsanity.com/paper/2504.11651/) (2025)
- [Stochastic Gradient Descent for Gaussian Processes Done Right](https://mlsanity.com/paper/2310.20581/) (2023)
- [FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models](https://mlsanity.com/paper/2501.10714/) (2025)
- [NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks](https://mlsanity.com/paper/2410.20650/) (2024)
- [ZipNN: Lossless Compression for AI Models](https://mlsanity.com/paper/2411.05239/) (2024)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs](https://mlsanity.com/paper/1903.02596/) (2019)
- [Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades](https://mlsanity.com/paper/2501.10661/) (2025)
- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference](https://mlsanity.com/paper/2502.16927/) (2025)
- [Lossless Compression for LLM Tensor Incremental Snapshots](https://mlsanity.com/paper/2505.09810/) (2025)
- [HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap](https://mlsanity.com/paper/2508.09591/) (2025)
- [UCCL-Zip: Lossless Compression Supercharged GPU Communication](https://mlsanity.com/paper/2604.17172/) (2026)

## Built on by (most cited first, 1 of 1)

- [CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Training](https://mlsanity.com/paper/2609.04609/) (2026)

## Abstract (arXiv)

Communication has emerged as a critical bottleneck in the distributed training of large language models (LLMs). While numerous approaches have been proposed to reduce communication overhead, the potential of lossless compression has remained largely underexplored since compression and decompression typically consume larger overheads than the benefits of reduced communication traffic. We observe that the communication data, including activations, gradients and parameters, during training often follows a near-Gaussian distribution, which is a key feature for data compression. Thus, we introduce ZipCCL, a lossless compressed communication library of collectives for LLM training. ZipCCL is equipped with our novel techniques: (1) theoretically grounded exponent coding that exploits the Gaussian distribution of LLM tensors to accelerate compression without expensive online statistics, (2) GPU-optimized compression and decompression kernels that carefully design memory access patterns and pipeline using communication-aware data layout, and (3) adaptive communication strategies that dynamically switch collective operations based on workload patterns and system characteristics. Evaluated on a 64-GPU cluster using both mixture-of-experts and dense transformer models, ZipCCL reduces communication time by up to 1.35× and achieves end-to-end training speedups of up to 1.18× without any impact on model quality.