mlsanityData to 14 Oct 2026

Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design

arXiv 2403.16125 ↗First posted 24 Mar 2024cs.DC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Efficiently training large-scale models (LMs) in GPU clusters involves two separate avenues: inter-job dynamic scheduling and intra-job adaptive parallelism (AP). However, existing dynamic schedulers struggle with large-model scheduling due to the mismatch between static parallelism (SP)-aware scheduling and AP-based execution, leading to cluster inefficiencies such as degraded throughput and prolonged job queuing. This paper presents Arena, a large-model training system that co-designs dynamic scheduling and adaptive parallelism to achieve high cluster efficiency. To reduce scheduling costs while improving decision quality, Arena designs low-cost, disaggregated profiling and AP-tailored, load-aware performance estimation, while unifying them by sharding the joint scheduling-parallelism optimization space via a grid abstraction. Building on this, Arena dynamically schedules profiled jobs in elasticity and heterogeneity dimensions, and executes them using efficient AP with pruned search space. Evaluated on heterogeneous testbeds and production workloads, Arena reduces job completion time by up to 49.3% and improves cluster throughput by up to 1.60×.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 75 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

26 of 75 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2016Wide Residual Networkscited by 3,070
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
14 more
  1. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  2. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  3. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  4. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  5. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  6. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  7. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  8. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  9. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  10. 2020Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Trainingcited by 22
  11. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 16
  12. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  13. 2023DynaPipe: Optimizing Multi-task Training through Dynamic Pipelinescited by 12
  14. 2023Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Searchcited by 2

Built on by

Not cited yet.