mlsanityData to 7 Oct 2026

Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters

arXiv 2307.12169 ↗First posted 22 Jul 2023cs.NI cs.AI cs.LGcited by 13

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

This paper presents a low-cost network architecture for training large language models (LLMs) at hyperscale. We study the optimal parallelization strategy of LLMs and propose a novel datacenter network design tailored to LLM's unique communication pattern. We show that LLM training generates sparse communication patterns in the network and, therefore, does not require any-to-any full-bisection network to complete efficiently. As a result, our design eliminates the spine layer in traditional GPU clusters. We name this design a Rail-only network and demonstrate that it achieves the same training performance while reducing the network cost by 38% to 77% and network power consumption by 37% to 75% compared to a conventional GPU datacenter. Our architecture also supports Mixture-of-Expert (MoE) models with all-to-all communication through forwarding, with only 8.2% to 11.2% completion time overhead for all-to-all traffic. We study the failure robustness of Rail-only networks and provide insights into the performance impact of different network and training parameters.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 12 of its 45 references that are in the map. First posted on arXiv 22 Jul 2023.

Builds on

12 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  7. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  8. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  9. 2022TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobscited by 30
  10. 2019Expanding across time to deliver bandwidth efficiency and low latencycited by 7
  11. 2023Bandwidth Optimal Pipeline Schedule for Collective Communicationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
  2. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  3. 2025InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceiverscited by 7
  4. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5
  5. 2024ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabricscited by 2
  6. 2024Exploring Scaling Laws for Local SGD in Large Language Model Trainingcited by 2
  7. 2025Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphscited by 1
  8. 2026Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inferencecited by 0
  9. 2026Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Servingcited by 0
  10. 2026Resilient AI Supercomputer Networking using MRC and SRv6cited by 0
  11. 2026Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatorycited by 0
  12. 2026mKernel: Fast Multi-GPU, Multi-Node Fused Kernelscited by 0
1 more
  1. 2026The Life of a Token: from Words to Bits on the Wirecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗
  2. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
  3. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
  4. 2025System-performance and cost modeling of Large Language Model training and inference
  5. 2025Scaling Intelligence: Designing Data Centers for Next-Gen Language Models
  6. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
  7. 2026Lodestar: An Online-Learning LLM Inference Router
  8. 2026Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer