mlsanityData to 7 Oct 2026

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

arXiv 2111.04007 ↗First posted 7 Nov 2021cs.DCcited by 40

Not on the map. Why?

Abstract (arXiv)

Systems for training massive deep learning models (billions of parameters) today assume and require specialized "hyper-clusters": hundreds or thousands of GPUs wired with specialized high-bandwidth interconnects such as NV-Link and Infiniband. Besides being expensive, such dependence on hyper-clusters and custom high-speed inter-connects limits the size of such clusters, creating (a) scalability limits on job parallelism; (b) resource fragmentation across hyper-clusters. In this paper, we present Varuna, a new system that enables training massive deep learning models on commodity networking. Varuna makes thrifty use of networking resources and automatically configures the user's training job to efficiently use any given set of resources. Therefore, Varuna is able to leverage "low-priority" VMs that cost about 5x cheaper than dedicated GPUs, thus significantly reducing the cost of training massive models. We demonstrate the efficacy of Varuna by training massive models, including a 200 billion parameter model, on 5x cheaper "spot VMs", while maintaining high training throughput. Varuna improves end-to-end training time by up to 18x compared to other model-parallel approaches and up to 26% compared to other pipeline parallel approaches. The code for Varuna is available at https://github.com/microsoft/varuna.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Nov 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 7; 2024: 13; 2025: 9; 2026: 5.

40 papers.

Sort:

By area:

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  3. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  4. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  7. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  8. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  9. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  10. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 23
  11. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  12. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
28 more
  1. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  2. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 15
  3. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  4. 2025Robust LLM Training Infrastructure at ByteDancecited by 14
  5. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  6. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  7. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  8. 2025Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checkscited by 10
  9. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  10. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 8
  11. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 6
  12. 2022Survey on Large Scale Neural Network Trainingcited by 4
  13. 2024Large Scale Generative AI Text Applied to Sports and Musiccited by 4
  14. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  15. 2025OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Trainingcited by 3
  16. 2023Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Searchcited by 2
  17. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 2
  18. 2022Pit-Pattern Classification of Colorectal Cancer Polyps Using a Hyper Sensitive Vision-Based Tactile Sensor and Dilated Residual Networkscited by 1
  19. 2023Exploring the Robustness of Decentralized Training for Large Language Modelscited by 1
  20. 2024Practical Forecasting of Cryptocoins Timeseries using Correlation Patternscited by 1
  21. 2025iServe: An Intent-based Serving System for LLMscited by 1
  22. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  23. 2026DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providerscited by 1
  24. 2024Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Trainingcited by 0
  25. 2026From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUscited by 0
  26. 2026PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpointcited by 0
  27. 2026When Is Availability-Aware Training Worth It? A Benchmark and Empirical Study of Interruption-Resilient Optimization Under Predictable Compute Schedulescited by 0
  28. 2026Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointingcited by 0