# SkyServe: Serving AI Models across Regions and Clouds with Spot Instances

arXiv: https://arxiv.org/abs/2411.01438 · HTML: https://mlsanity.com/paper/2411.01438/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 70 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [SpotServe: Serving Generative Large Language Models on Preemptible Instances](https://mlsanity.com/paper/2311.15566/) (2023)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [Varuna: Scalable, Low-cost Training of Massive Deep Learning Models](https://mlsanity.com/paper/2111.04007/) (2021)
- [Serving deep learning models in a serverless platform](https://mlsanity.com/paper/1710.08460/) (2017)
- [BARISTA: Efficient and Scalable Serverless Serving System for Deep Learning Prediction Services](https://mlsanity.com/paper/1904.01576/) (2019)
- [A Reliable and Cost-Efficient Auto-Scaling System for Web Applications Using Heterogeneous Spot Instances](https://mlsanity.com/paper/1509.05197/) (2015)

## Built on by (most cited first, 20 of 20)

- [Barbarians at the Gate: How AI is Upending Systems Research](https://mlsanity.com/paper/2510.06189/) (2025)
- [UCCL-EP: Portable Expert-Parallel Communication](https://mlsanity.com/paper/2512.19849/) (2025)
- [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2505.11415/) (2025)
- [RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs](https://mlsanity.com/paper/2510.19225/) (2025)
- [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](https://mlsanity.com/paper/2511.00796/) (2025)
- [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2602.10729/) (2026)
- [ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching](https://mlsanity.com/paper/2509.16857/) (2025)
- [Let the Barbarians In: How AI Can Accelerate Systems Performance Research](https://mlsanity.com/paper/2512.14806/) (2025)
- [Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs](https://mlsanity.com/paper/2605.04357/) (2026)
- [Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling](https://mlsanity.com/paper/2507.18006/) (2025)
- [Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictions](https://mlsanity.com/paper/2512.20967/) (2025)
- [Efficient Remote KV Cache Reuse with GPU-native Video Codec](https://mlsanity.com/paper/2602.09725/) (2026)
- [GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving](https://mlsanity.com/paper/2602.11688/) (2026)
- [Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics](https://mlsanity.com/paper/2604.07144/) (2026)
- [ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters](https://mlsanity.com/paper/2606.18600/) (2026)
- [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2412.07067/) (2024)
- [HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling](https://mlsanity.com/paper/2605.16637/) (2026)
- [Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training](https://mlsanity.com/paper/2606.19004/) (2026)
- [Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs](https://mlsanity.com/paper/2607.23115/) (2026)
- [ExaServe: Large-Scale LLM Serving on Exascale HPC Systems](https://mlsanity.com/paper/2609.10812/) (2026)

## Abstract (arXiv)

Recent years have witnessed an explosive growth of AI models. The high cost of hosting AI services on GPUs and their demanding service requirements, make it timely and challenging to lower service costs and guarantee service quality. While spot instances have long been offered with a large discount, spot preemptions have discouraged users from using them to host model replicas when serving AI models. To address this, we propose a simple yet efficient policy, SpotHedge, that leverages spot replicas across different failure domains (e.g., regions and clouds) to ensure availability, lower costs, and high service quality. SpotHedge intelligently spreads spot replicas across different regions and clouds to improve availability and reduce correlated preemptions, overprovisions cheap spot replicas than required as a safeguard against possible preemptions, and dynamically falls back to on-demand replicas when spot replicas become unavailable. We built SkyServe, a system leveraging SpotHedge to efficiently serve AI models over a mixture of spot and on-demand replicas across regions and clouds. We compared SkyServe with both research and production systems on real AI workloads: SkyServe reduces cost by 43% on average while achieving high resource availability compared to using on-demand replicas. Additionally, SkyServe improves P50, P90, and P99 latency by 2.3×, 2.1×, 2.1× on average compared to other research and production systems.