mlsanityData to 7 Oct 2026

SpotServe: Serving Generative Large Language Models on Preemptible Instances

arXiv 2311.15566 ↗First posted 27 Nov 2023cs.DC cs.CL cs.LGcited by 83

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPUs at a much cheaper price than regular instances but may be preempted by the cloud at any time. Serving LLMs on preemptible instances requires addressing challenges induced by frequent instance preemptions and the necessity of migrating instances to handle these preemptions. This paper presents SpotServe, the first distributed LLM serving system on preemptible instances. Several key techniques in SpotServe realize fast and reliable serving of generative LLMs on cheap preemptible instances. First, SpotServe dynamically adapts the LLM parallelization configuration for dynamic instance availability and fluctuating workload, while balancing the trade-off among the overall throughput, inference latency and monetary costs. Second, to minimize the cost of migrating instances for dynamic reparallelization, the task of migrating instances is formulated as a bipartite graph matching problem, which uses the Kuhn-Munkres algorithm to identify an optimal migration plan that minimizes communications. Finally, to take advantage of the grace period offered by modern clouds, we introduce stateful inference recovery, a new inference mechanism that commits inference progress at a much finer granularity and allows SpotServe to cheaply resume inference upon preemption. We evaluate on real spot instance preemption traces and various popular LLMs and show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 60 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

20 of 60 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  11. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  12. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
8 more
  1. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  2. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  3. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  4. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  5. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  6. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  7. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  8. 2020Optimizing Prediction Serving on Low-Latency Serverless Dataflowcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 30; 2026: 27.

83 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  3. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  7. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
  8. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  9. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  10. 2024Queue management for slo-oriented large language model servingcited by 36
  11. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  12. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
38 more
  1. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  2. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  3. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  4. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  5. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  6. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  7. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 18
  8. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  9. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  10. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  11. 2024Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inferencecited by 13
  12. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Servingcited by 13
  13. 2025Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptationcited by 12
  14. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  15. 2026Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxescited by 11
  16. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  17. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMindcited by 10
  18. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  19. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  20. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 9
  21. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  22. 2025On Evaluating Performance of LLM Inference Serving Systemscited by 9
  23. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictorcited by 8
  24. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  25. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  26. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  27. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  28. 2024BlockLLM: Multi-tenant Finer-grained Serving for Large Language Modelscited by 5
  29. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5
  30. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  31. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  32. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
  33. 2025TetriServe: Efficiently Serving Mixed DiT Workloadscited by 4
  34. 2024MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programmingcited by 3
  35. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3
  36. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  37. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructurecited by 3
  38. 2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MoLink: Distributed and Efficient Serving Framework for Large Models
  2. 2025Characterizing Communication Patterns in Distributed Large Language Model Inference
  3. 2023Fast Distributed Inference Serving for Large Language Models
  4. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
  5. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
  7. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  8. 2025Taming the Titans: A Survey of Efficient LLM Inference Serving