# Blink: Fast and Generic Collectives for Distributed ML

arXiv: https://arxiv.org/abs/1910.04940 · HTML: https://mlsanity.com/paper/1910.04940/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 30 of 36)

- [Accelerating Distributed MoE Training and Inference with Lina](https://mlsanity.com/paper/2210.17223/) (2022)
- [Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training](https://mlsanity.com/paper/2208.06102/) (2022)
- [ZeRO++: Extremely Efficient Collective Communication for Giant Model Training](https://mlsanity.com/paper/2306.10209/) (2023)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters](https://mlsanity.com/paper/2308.00852/) (2023)
- [On Optimizing the Communication of Model Parallelism](https://mlsanity.com/paper/2211.05322/) (2022)
- [Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models](https://mlsanity.com/paper/2110.04478/) (2021)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping](https://mlsanity.com/paper/2409.15241/) (2024)
- [Impact of RoCE Congestion Control Policies on Distributed Training of DNNs](https://mlsanity.com/paper/2207.10898/) (2022)
- [Efficient Direct-Connect Topologies for Collective Communications](https://mlsanity.com/paper/2202.03356/) (2022)
- [PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices](https://mlsanity.com/paper/2310.19991/) (2023)
- [Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective](https://mlsanity.com/paper/2509.10371/) (2025)
- [Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning](https://mlsanity.com/paper/2110.10548/) (2021)
- [TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning](https://mlsanity.com/paper/2304.05301/) (2023)
- [Rethinking Memory and Communication Cost for Efficient Large Language Model Training](https://mlsanity.com/paper/2310.06003/) (2023)
- [LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models](https://mlsanity.com/paper/2109.11762/) (2021)
- [DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining](https://mlsanity.com/paper/2302.12445/) (2023)
- [Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities](https://mlsanity.com/paper/2403.07585/) (2024)
- [ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics](https://mlsanity.com/paper/2402.06787/) (2024)
- [MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training](https://mlsanity.com/paper/2103.15195/) (2021)
- [EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks](https://mlsanity.com/paper/2110.09132/) (2021)
- [ByteComp: Revisiting Gradient Compression in Distributed Training](https://mlsanity.com/paper/2205.14465/) (2022)
- [Efficient All-reduce for Distributed DNN Training in Optical Interconnect System](https://mlsanity.com/paper/2207.10982/) (2022)
- [Bandwidth Optimal Pipeline Schedule for Collective Communication](https://mlsanity.com/paper/2305.18461/) (2023)
- [Efficient AllReduce with Stragglers](https://mlsanity.com/paper/2505.23523/) (2025)
- [LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIM](https://mlsanity.com/paper/2604.04523/) (2026)
- [ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training](https://mlsanity.com/paper/2604.27844/) (2026)
- [Variant Parallelism: Lightweight Deep Convolutional Models for Distributed Inference on IoT Devices](https://mlsanity.com/paper/2210.08376/) (2022)

## Abstract (arXiv)

Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale. Existing parameter synchronization protocols cannot effectively leverage available network resources in the face of ever increasing hardware heterogeneity. To address this, we propose Blink, a collective communication library that dynamically generates optimal communication primitives by packing spanning trees. We propose techniques to minimize the number of trees generated and extend Blink to leverage heterogeneous communication channels for faster data transfers. Evaluations show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8x faster model synchronization, and reduce end-to-end training time for image classification tasks by up to 40%.