# Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters

arXiv: https://arxiv.org/abs/2504.17096 · HTML: https://mlsanity.com/paper/2504.17096/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2025.

## Built on by (most cited first, 13 of 13)

- [Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving](https://mlsanity.com/paper/2512.14098/) (2025)
- [Decoupled DiLoCo for Resilient Distributed Pre-training](https://mlsanity.com/paper/2604.21428/) (2026)
- [Libra: Efficient Resource Management for Agentic RL Post-Training](https://mlsanity.com/paper/2606.03077/) (2026)
- [tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Models](https://mlsanity.com/paper/2602.07263/) (2026)
- [Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs](https://mlsanity.com/paper/2605.04357/) (2026)
- [Dora: QoE-Aware Hybrid Parallelism for Distributed Edge AI](https://mlsanity.com/paper/2512.10990/) (2025)
- [Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation](https://mlsanity.com/paper/2604.10180/) (2026)
- [ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training](https://mlsanity.com/paper/2605.24326/) (2026)
- [Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving](https://mlsanity.com/paper/2607.16488/) (2026)
- [ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork](https://mlsanity.com/paper/2608.23840/) (2026)
- [A principled approach for energy-efficient training via phase-aware GPU frequency tuning](https://mlsanity.com/paper/2609.24205/) (2026)
- [HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training](https://mlsanity.com/paper/2609.39350/) (2026)
- [ATTUNE: Agentic Throughput Tuning for LLM Training](https://mlsanity.com/paper/2610.17528/) (2026)

## Abstract (arXiv)

The high GPU demand of ML training makes it hard to allocate large homogeneous clusters of high-end GPUs in a single availability zone. Leveraging heterogeneous GPUs available within and across zones can improve throughput at a reasonable cost. However, training ML models on heterogeneous resources introduces significant challenges, such as stragglers and a large search space of possible job configurations. Current systems lack support for efficiently training models on heterogeneous resources. We present Sailor, a system that automates distributed training over heterogeneous, geo-distributed, and dynamically available resources. Sailor combines an efficient search space exploration algorithm, accurate runtime and memory footprint simulation, and a distributed training framework that supports different types of heterogeneity to optimize training throughput and cost.