mlsanityData to 14 Oct 2026

Routing for Large ML Models

arXiv 2503.05324 ↗First posted 7 Mar 2025cs.NI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Training large language models (LLMs), and other large machine learning models, involves repeated communication of large volumes of data across a data center network. The communication patterns induced by these training process exhibit high regularity and persistence, giving rise to significant opportunities for optimizing the manner in which flows are routed across the network. We present an algorithmic framework for textitquantifying network-wide efficiency in the context of training LLMs (and other large-scale ML models), and for periodically textitoptimizing routing with respect to this global metric.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 50 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

19 of 50 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  8. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  9. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  10. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  11. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  12. 2020Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systemscited by 38
7 more
  1. 2019Blink: Fast and Generic Collectives for Distributed MLcited by 36
  2. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  3. 2018TicTac: Accelerating Distributed Deep Learning with Communication Schedulingcited by 27
  4. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
  5. 2015FireCaffe: near-linear acceleration of deep neural network training on compute clusterscited by 25
  6. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  7. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21

Built on by

Not cited yet.