mlsanityData to 14 Oct 2026

How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study

arXiv 2306.03163 ↗First posted 5 Jun 2023cs.LG cs.DC cs.NIcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper aims to answer the question: Can deep learning models be cost-efficiently trained on a global market of spot VMs spanning different data centers and cloud providers? To provide guidance, we extensively evaluate the cost and throughput implications of training in different zones, continents, and clouds for representative CV, NLP, and ASR models. To expand the current training options further, we compare the scalability potential for hybrid-cloud scenarios by adding cloud resources to on-premise hardware to improve training throughput. Finally, we show how leveraging spot instance pricing enables a new cost-efficient way to train models with multiple cheap VMs, trumping both more centralized and powerful hardware and even on-demand cloud offerings at competitive prices.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 44 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

20 of 44 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  9. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  10. 2019MLPerf Training Benchmarkcited by 115
  11. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  12. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
8 more
  1. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  2. 2021Distributed Deep Learning in Open Collaborationscited by 58
  3. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  4. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  5. 2020High Performance I/O For Large Scale Deep Learningcited by 26
  6. 2020Amazon SageMaker Autopilot: a white box AutoML solution at scalecited by 20
  7. 2022Training Transformers Togethercited by 5
  8. 2022SpotLake: Diverse Spot Instance Dataset Archive Servicecited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
  2. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  3. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 4
  4. 2024Towards providing reliable job completion time predictions using PCScited by 0