mlsanityData to 7 Oct 2026

Efficient Direct-Connect Topologies for Collective Communications

arXiv 2202.03356 ↗First posted 7 Feb 2022cs.NI cs.DC cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We consider the problem of distilling efficient network topologies for collective communications. We provide an algorithmic framework for constructing direct-connect topologies optimized for the latency vs. bandwidth trade-off associated with the workload. Our approach synthesizes many different topologies and schedules for a given cluster size and degree and then identifies the appropriate topology and schedule for a given workload. Our algorithms start from small, optimal base topologies and associated communication schedules and use techniques that can be iteratively applied to derive much larger topologies and schedules. Additionally, we incorporate well-studied large-scale graph topologies into our algorithmic framework by producing efficient collective schedules for them using a novel polynomial-time algorithm. Our evaluation uses multiple testbeds and large-scale simulations to demonstrate significant performance benefits from our derived topologies and schedules.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 116 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

26 of 116 references.

References · 26

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  9. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  10. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  11. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  12. 2017Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusterscited by 55
14 more
  1. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  2. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  3. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 37
  4. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 34
  5. 2017AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Trainingcited by 33
  6. 2020Synthesizing Optimal Collective Algorithmscited by 30
  7. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  8. 2018Double Quantization for Communication-Efficient Distributed Optimizationcited by 18
  9. 2019Expanding across time to deliver bandwidth efficiency and low latencycited by 7
  10. 2019Slim Fly: A Cost Effective Low-Diameter Network Topologycited by 7
  11. 2022GC3: An Optimizing Compiler for GPU Collective Communicationcited by 6
  12. 2020Srifty: Swift and Thrifty Distributed Training on the Cloudcited by 4
  13. 2016A Survey of Methods for Collective Communication Optimization and Tuningcited by 2
  14. 2021SpectralFly: Ramanujan Graphs as Flexible and Efficient Interconnection Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  2. 2025Robust Heuristic Algorithm Design with LLMscited by 3
  3. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  4. 2025Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networkscited by 0
  5. 2026Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANscited by 0