mlsanityData to 7 Oct 2026

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

arXiv 2202.00433 ↗First posted 1 Feb 2022cs.NI cs.DCcited by 30

Not on the map. Why?

Abstract (arXiv)

We propose TopoOpt, a novel direct-connect fabric for deep neural network (DNN) training workloads. TopoOpt co-optimizes the distributed training process across three dimensions: computation, communication, and network topology. We demonstrate the mutability of AllReduce traffic, and leverage this property to construct efficient network topologies for DNN training jobs. TopoOpt then uses an alternating optimization technique and a group theory-inspired algorithm called TotientPerms to find the best network topology and routing plan, together with a parallelization strategy. We build a fully functional 12-node direct-connect prototype with remote direct memory access (RDMA) forwarding at 100 Gbps. Large-scale simulations on real distributed training models show that, compared to similar-cost Fat-Tree interconnects, TopoOpt reduces DNN training time by up to 3.4x.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Feb 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 8; 2025: 10; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  2. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  3. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  4. 2024Forecasting GPU Performance for Deep Learning Training and Inferencecited by 15
  5. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  6. 2023Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameterscited by 13
  7. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  8. 2024Swing: Short-cutting Rings for Higher Bandwidth Allreducecited by 8
  9. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
  10. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7
  11. 2022Efficient Direct-Connect Topologies for Collective Communicationscited by 5
  12. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
18 more
  1. 2023TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learningcited by 4
  2. 2023Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environmentcited by 4
  3. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swapcited by 4
  4. 2024Communication Optimization for Distributed Training: Architecture, Advances, and Opportunitiescited by 3
  5. 2025Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learningcited by 3
  6. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  7. 2024Credence: Augmenting Datacenter Switch Buffer Sharing with ML Predictionscited by 1
  8. 2025Efficient AllReduce with Stragglerscited by 1
  9. 2026NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inferencecited by 1
  10. 2024MLCC: A Congestion Control Technique to Accelerate ML Trainingcited by 0
  11. 2024VcLLM: Video Codecs are Secretly Tensor Codecscited by 0
  12. 2025Routing for Large ML Modelscited by 0
  13. 2025LUMION: Fast Fault Recovery for ML Jobs Using Programmable Optical Fabricscited by 0
  14. 2025Morphlux: Transforming Torus Fabrics for Efficient Multi-tenant MLcited by 0
  15. 2025Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networkscited by 0
  16. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
  17. 2026Step-TP: A Grounded, Step-Level Dataset with Chain-of-Thought Reasoning for LLM-Guided Tensor Program Optimizationcited by 0
  18. 2026AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformerscited by 0