# TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

arXiv: https://arxiv.org/abs/2202.00433 · HTML: https://mlsanity.com/paper/2202.00433/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2022.

## Built on by (most cited first, 30 of 30)

- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters](https://mlsanity.com/paper/2308.00852/) (2023)
- [THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression](https://mlsanity.com/paper/2302.08545/) (2023)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems](https://mlsanity.com/paper/2310.02784/) (2023)
- [Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters](https://mlsanity.com/paper/2307.12169/) (2023)
- [Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2404.06114/) (2024)
- [Swing: Short-cutting Rings for Higher Bandwidth Allreduce](https://mlsanity.com/paper/2401.09356/) (2024)
- [MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training](https://mlsanity.com/paper/2501.03905/) (2025)
- [InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers](https://mlsanity.com/paper/2502.03885/) (2025)
- [Efficient Direct-Connect Topologies for Collective Communications](https://mlsanity.com/paper/2202.03356/) (2022)
- [Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective](https://mlsanity.com/paper/2509.10371/) (2025)
- [TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning](https://mlsanity.com/paper/2304.05301/) (2023)
- [Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment](https://mlsanity.com/paper/2312.03549/) (2023)
- [HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap](https://mlsanity.com/paper/2508.09591/) (2025)
- [Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities](https://mlsanity.com/paper/2403.07585/) (2024)
- [Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learning](https://mlsanity.com/paper/2504.20854/) (2025)
- [ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics](https://mlsanity.com/paper/2402.06787/) (2024)
- [Credence: Augmenting Datacenter Switch Buffer Sharing with ML Predictions](https://mlsanity.com/paper/2401.02801/) (2024)
- [Efficient AllReduce with Stragglers](https://mlsanity.com/paper/2505.23523/) (2025)
- [NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference](https://mlsanity.com/paper/2606.03910/) (2026)
- [MLCC: A Congestion Control Technique to Accelerate ML Training](https://mlsanity.com/paper/2402.09589/) (2024)
- [VcLLM: Video Codecs are Secretly Tensor Codecs](https://mlsanity.com/paper/2407.00467/) (2024)
- [Routing for Large ML Models](https://mlsanity.com/paper/2503.05324/) (2025)
- [LUMION: Fast Fault Recovery for ML Jobs Using Programmable Optical Fabrics](https://mlsanity.com/paper/2505.23105/) (2025)
- [Morphlux: Transforming Torus Fabrics for Efficient Multi-tenant ML](https://mlsanity.com/paper/2508.03674/) (2025)
- [Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networks](https://mlsanity.com/paper/2510.19322/) (2025)
- [Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving](https://mlsanity.com/paper/2605.00254/) (2026)
- [Step-TP: A Grounded, Step-Level Dataset with Chain-of-Thought Reasoning for LLM-Guided Tensor Program Optimization](https://mlsanity.com/paper/2605.25954/) (2026)
- [AoiZora: Topology-Aware Auto-Parallel Optimization for Inference of Diffusion Transformers](https://mlsanity.com/paper/2606.17566/) (2026)

## Abstract (arXiv)

We propose TopoOpt, a novel direct-connect fabric for deep neural network (DNN) training workloads. TopoOpt co-optimizes the distributed training process across three dimensions: computation, communication, and network topology. We demonstrate the mutability of AllReduce traffic, and leverage this property to construct efficient network topologies for DNN training jobs. TopoOpt then uses an alternating optimization technique and a group theory-inspired algorithm called TotientPerms to find the best network topology and routing plan, together with a parallelization strategy. We build a fully functional 12-node direct-connect prototype with remote direct memory access (RDMA) forwarding at 100 Gbps. Large-scale simulations on real distributed training models show that, compared to similar-cost Fat-Tree interconnects, TopoOpt reduces DNN training time by up to 3.4x.