mlsanityData to 7 Oct 2026

SkyServe: Serving AI Models across Regions and Clouds with Spot Instances

arXiv 2411.01438 ↗First posted 3 Nov 2024cs.DC cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed an explosive growth of AI models. The high cost of hosting AI services on GPUs and their demanding service requirements, make it timely and challenging to lower service costs and guarantee service quality. While spot instances have long been offered with a large discount, spot preemptions have discouraged users from using them to host model replicas when serving AI models. To address this, we propose a simple yet efficient policy, SpotHedge, that leverages spot replicas across different failure domains (e.g., regions and clouds) to ensure availability, lower costs, and high service quality. SpotHedge intelligently spreads spot replicas across different regions and clouds to improve availability and reduce correlated preemptions, overprovisions cheap spot replicas than required as a safeguard against possible preemptions, and dynamically falls back to on-demand replicas when spot replicas become unavailable. We built SkyServe, a system leveraging SpotHedge to efficiently serve AI models over a mixture of spot and on-demand replicas across regions and clouds. We compared SkyServe with both research and production systems on real AI workloads: SkyServe reduces cost by 43% on average while achieving high resource availability compared to using on-demand replicas. Additionally, SkyServe improves P50, P90, and P99 latency by 2.3×, 2.1×, 2.1× on average compared to other research and production systems.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 13 of its 70 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

13 of 70 references.

References · 13

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  6. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 83
  8. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  9. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  10. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  11. 2017Serving deep learning models in a serverless platformcited by 9
  12. 2019BARISTA: Efficient and Scalable Serverless Serving System for Deep Learning Prediction Servicescited by 6
1 more
  1. 2015A Reliable and Cost-Efficient Auto-Scaling System for Web Applications Using Heterogeneous Spot Instancescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 48
  2. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  3. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 11
  4. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  5. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  6. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  7. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  8. 2025Let the Barbarians In: How AI Can Accelerate Systems Performance Researchcited by 2
  9. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scalingcited by 1
  10. 2025Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictionscited by 1
  11. 2026GORGO: Online Tuning for Cross-Region Network-Aware LLM Servingcited by 1
  12. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
7 more
  1. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 1
  2. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusterscited by 1
  3. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 0
  4. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
  5. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  6. 2026Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANscited by 0
  7. 2026ExaServe: Large-Scale LLM Serving on Exascale HPC Systemscited by 0