# SparCML: High-Performance Sparse Communication for Machine Learning

arXiv: https://arxiv.org/abs/1802.08021 · HTML: https://mlsanity.com/paper/1802.08021/

## Key facts

- Cited by 34 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 22 Feb 2018.

## Built on by (most cited first, 30 of 34)

- [Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines](https://mlsanity.com/paper/2107.06925/) (2021)
- [Pufferfish: Communication-efficient Models At No Extra Cost](https://mlsanity.com/paper/2103.03936/) (2021)
- [Near-Optimal Sparse Allreduce for Distributed Deep Learning](https://mlsanity.com/paper/2201.07598/) (2022)
- [Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks](https://mlsanity.com/paper/2202.02947/) (2022)
- [Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients](https://mlsanity.com/paper/2406.17660/) (2024)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [HammingMesh: A Network Topology for Large-Scale Deep Learning](https://mlsanity.com/paper/2209.01346/) (2022)
- [Evaluation and Optimization of Gradient Compression for Distributed Deep Learning](https://mlsanity.com/paper/2306.08881/) (2023)
- [Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules](https://mlsanity.com/paper/2407.00599/) (2024)
- [CGX: Adaptive System Support for Communication-Efficient Deep Learning](https://mlsanity.com/paper/2111.08617/) (2021)
- [Reconciling Security and Communication Efficiency in Federated Learning](https://mlsanity.com/paper/2207.12779/) (2022)
- [Accelerating Distributed Deep Learning using Lossless Homomorphic Compression](https://mlsanity.com/paper/2402.07529/) (2024)
- [Efficient Strong Scaling Through Burst Parallel Training](https://mlsanity.com/paper/2112.10065/) (2021)
- [Sparse-SignSGD with Majority Vote for Communication-Efficient Distributed Learning](https://mlsanity.com/paper/2302.07475/) (2023)
- [DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning](https://mlsanity.com/paper/2102.03112/) (2021)
- [An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks](https://mlsanity.com/paper/2104.09075/) (2021)
- [DQRM: Deep Quantized Recommendation Models](https://mlsanity.com/paper/2410.20046/) (2024)
- [Noisy Truncated SGD: Optimization and Generalization](https://mlsanity.com/paper/2103.00075/) (2021)
- [EventGraD: Event-Triggered Communication in Parallel Machine Learning](https://mlsanity.com/paper/2103.07454/) (2021)
- [MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training](https://mlsanity.com/paper/2103.15195/) (2021)
- [S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning](https://mlsanity.com/paper/2110.02140/) (2021)
- [ByteComp: Revisiting Gradient Compression in Distributed Training](https://mlsanity.com/paper/2205.14465/) (2022)
- [L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning](https://mlsanity.com/paper/2210.17357/) (2022)
- [STen: Productive and Efficient Sparsity in PyTorch](https://mlsanity.com/paper/2304.07613/) (2023)
- [Lion Cub: Minimizing Communication Overhead in Distributed Lion](https://mlsanity.com/paper/2411.16462/) (2024)
- [Efficient AllReduce with Stragglers](https://mlsanity.com/paper/2505.23523/) (2025)
- [CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation](https://mlsanity.com/paper/2106.10796/) (2021)
- [Resource-Constrained Decentralized Federated Learning via Personalized Event-Triggering](https://mlsanity.com/paper/2211.12640/) (2022)
- [A Theory of I/O-Efficient Sparse Neural Network Inference](https://mlsanity.com/paper/2301.01048/) (2023)
- [SparDL: Distributed Deep Learning Training with Efficient Sparse Communication](https://mlsanity.com/paper/2304.00737/) (2023)

## Abstract (arXiv)

Applying machine learning techniques to the quickly growing data in science and industry requires highly-scalable algorithms. Large datasets are most commonly processed "data parallel" distributed across many nodes. Each node's contribution to the overall gradient is summed using a global allreduce. This allreduce is the single communication and thus scalability bottleneck for most machine learning workloads. We observe that frequently, many gradient values are (close to) zero, leading to sparse of sparsifyable communications. To exploit this insight, we analyze, design, and implement a set of communication-efficient protocols for sparse input data, in conjunction with efficient machine learning algorithms which can leverage these primitives. Our communication protocols generalize standard collective operations, by allowing processes to contribute arbitrary sparse input data vectors. Our generic communication library, SparCML, extends MPI to support additional features, such as non-blocking (asynchronous) operations and low-precision data representations. As such, SparCML and its techniques will form the basis of future highly-scalable machine learning frameworks.