mlsanityData to 14 Oct 2026

Blink: Fast and Generic Collectives for Distributed ML

arXiv 1910.04940 ↗First posted 11 Oct 2019cs.DC cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale. Existing parameter synchronization protocols cannot effectively leverage available network resources in the face of ever increasing hardware heterogeneity. To address this, we propose Blink, a collective communication library that dynamically generates optimal communication primitives by packing spanning trees. We propose techniques to minimize the number of trees generated and extend Blink to leverage heterogeneous communication channels for faster data transfers. Evaluations show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8x faster model synchronization, and reduce end-to-end training time for image classification tasks by up to 40%.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 8; 2024: 4; 2025: 4; 2026: 6.

36 papers.

Sort:

By area:

  1. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  2. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  3. 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
  4. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  5. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  6. 2022On Optimizing the Communication of Model Parallelismcited by 19
  7. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 16
  8. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 15
  9. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
  10. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 9
  11. 2022Impact of RoCE Congestion Control Policies on Distributed Training of DNNscited by 6
  12. 2022Efficient Direct-Connect Topologies for Collective Communicationscited by 5
24 more
  1. 2023PolyThrottle: Energy-efficient Neural Network Inference on Edge Devicescited by 5
  2. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  3. 2021Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learningcited by 4
  4. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  5. 2023Rethinking Memory and Communication Cost for Efficient Large Language Model Trainingcited by 4
  6. 2021LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Modelscited by 3
  7. 2023DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipeliningcited by 3
  8. 2024Communication Optimization for Distributed Training: Architecture, Advances, and Opportunitiescited by 3
  9. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  10. 2021MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Trainingcited by 1
  11. 2021EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networkscited by 1
  12. 2022ByteComp: Revisiting Gradient Compression in Distributed Trainingcited by 1
  13. 2022Efficient All-reduce for Distributed DNN Training in Optical Interconnect Systemcited by 1
  14. 2023Bandwidth Optimal Pipeline Schedule for Collective Communicationcited by 1
  15. 2025Efficient AllReduce with Stragglerscited by 1
  16. 2026LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIMcited by 1
  17. 2026ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Trainingcited by 1
  18. 2022Variant Parallelism: Lightweight Deep Convolutional Models for Distributed Inference on IoT Devicescited by 0
  19. 2025Routing for Large ML Modelscited by 0
  20. 2025Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networkscited by 0
  21. 2026Don't Let a Few Network Failures Slow the Entire AllReducecited by 0
  22. 2026ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modelingcited by 0
  23. 2026NEURON-Fabric: Architecture-Runtime Co-Design for Controlled Low-Bit Gradient Communicationcited by 0
  24. 2026Topology-Aware Data Movement for Disaggregated GPU Inferencecited by 0