mlsanityData to 7 Oct 2026

TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches

arXiv 2111.04867 ↗First posted 8 Nov 2021cs.DC cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Machine learning models are increasingly being trained across multiple GPUs and servers. In this setting, data is transferred between GPUs using communication collectives such as AlltoAll and AllReduce, which can become a significant bottleneck in training large models. Thus, it is important to use efficient algorithms for collective communication. We develop TACCL, a tool that enables algorithm designers to guide a synthesizer into automatically generating algorithms for a given hardware configuration and communication collective. TACCL uses a novel communication sketch abstraction to get crucial information from the designer to significantly reduce the search space and guide the synthesizer towards better algorithms. TACCL also uses a novel encoding of the problem that allows it to scale beyond single-node topologies. We use TACCL to synthesize algorithms for three collectives and two hardware topologies: DGX-2 and NDv2. We demonstrate that the algorithms synthesized by TACCL outperform the Nvidia Collective Communication Library (NCCL) by up to 6.7x. We also show that TACCL can speed up end-to-end training of Transformer-XL and BERT models by 11%--2.3x for different batch sizes.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 15 of its 51 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

15 of 51 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  7. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  8. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  9. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 62
  10. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 34
  11. 2020Synthesizing Optimal Collective Algorithmscited by 30
  12. 2020Is Network the Bottleneck of Distributed Training?cited by 19
3 more
  1. 2018Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Trainingcited by 17
  2. 2021Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learningcited by 4
  3. 2015JSKETCH: Sketching for Javacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 9; 2025: 12; 2026: 12.

38 papers.

Sort:

By area:

  1. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  2. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismcited by 31
  3. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  4. 2024T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectivescited by 15
  5. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  6. 2025MSCCL++: Rethinking GPU Communication Abstractions for AI Inferencecited by 9
  7. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  8. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 8
  9. 2022Efficient Direct-Connect Topologies for Collective Communicationscited by 5
  10. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  11. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  12. 2024Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimizationcited by 4
26 more
  1. 2025Collective Communication for 100k+ GPUscited by 4
  2. 2025Reliable and Resilient Collective Communication Library for LLM Training and Servingcited by 4
  3. 2026UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Trainingcited by 4
  4. 2024Communication Optimization for Distributed Training: Architecture, Advances, and Opportunitiescited by 3
  5. 2025Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learningcited by 3
  6. 2025Robust Heuristic Algorithm Design with LLMscited by 3
  7. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  8. 2023Bandwidth Optimal Pipeline Schedule for Collective Communicationcited by 1
  9. 2023Network Contention-Aware Cluster Scheduling with Reinforcement Learningcited by 1
  10. 2025Efficient AllReduce with Stragglerscited by 1
  11. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  12. 2026GF-DiT: Scheduling Parallelism for Diffusion Transformer Servingcited by 1
  13. 2024MLCC: A Congestion Control Technique to Accelerate ML Trainingcited by 0
  14. 2025GPU-centric Communication Schemes for HPC and ML Applicationscited by 0
  15. 2025LUMION: Fast Fault Recovery for ML Jobs Using Programmable Optical Fabricscited by 0
  16. 2025Morphlux: Transforming Torus Fabrics for Efficient Multi-tenant MLcited by 0
  17. 2025Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networkscited by 0
  18. 2026FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Costcited by 0
  19. 2026HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusterscited by 0
  20. 2026Don't Let a Few Network Failures Slow the Entire AllReducecited by 0
  21. 2026AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformerscited by 0
  22. 2026Distributed Training using an Intelligent Networkcited by 0
  23. 2026Tuning Collective Patterns to Alleviate Congestion in Shared AI Clusterscited by 0
  24. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processingcited by 0
  25. 2026mKernel: Fast Multi-GPU, Multi-Node Fused Kernelscited by 0
  26. 2026Purlin: Separating Orchestration from the Datapath of Collectivescited by 0