Improving training time and GPU utilization in geo-distributed language model training
Not on the map. Why?
Abstract (arXiv)
The widespread adoption of language models (LMs) has caused a huge surge in demand for GPUs. Training large LMs requires tens of thousands of GPUs and housing them in the same datacenter (DC) is a challenge due to many constraints including availability of peak power. We focus on training such models across multiple DCs connected via the Wide-Area-Network (WAN). We built Atlas that speeds up the training time using novel workload-aware temporal bandwidth sharing and other design choices. While Atlas improves the training time, it does not completely eliminate the bubbles (idle GPU cycles). We built BubbleTea that runs prefill-as-a-service (part of LM inference) during the bubbles thus improving the GPU utilization without any impact on training. Compared to state-of-the-art designs, Atlas and BubbleTea together achieve up to 17x faster training, and up to 94% GPU utilization. The code will be open-sourced.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 85 references that are in the map. First posted on arXiv 16 Nov 2024.
Builds on
34 of 85 references.
References · 34
- 2017Attention Is All You Needcited by 60,361
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
- 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
- 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
22 more
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
- 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
- 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
- 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
- 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
- 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
- 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
- 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
- 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
- 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
- 2023Zero Bubble Pipeline Parallelismcited by 40
- 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26
- 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
- 2020Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusterscited by 23
- 2023FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUscited by 20
- 2021Tesseract: Parallelize the Tensor Parallelism Efficientlycited by 19
- 2021Communication-efficient SGD: From Local SGD to One-Shot Averagingcited by 19
- 2022On Distributed Adaptive Optimization with Gradient Compressioncited by 19
- 2023Does compressing activations help model parallel training?cited by 12
- 2023Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problemcited by 11
- 2022Skyplane: Optimizing Transfer Cost and Throughput Using Cloud-Aware Overlayscited by 5
- 2024PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Trainingcited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.