mlsanityData to 14 Oct 2026

ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters

arXiv 2606.18600 ↗First posted 17 Jun 2026cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As large language model (LLM) services become widely adopted, the cost of GPU resources for serving these models in cloud environments has emerged as a critical concern. Spot instances offer up to 90% cost savings over on-demand instances, but their frequent interruptions and limited availability pose significant challenges for continuous LLM serving. GPU spot instances, in particular, exhibit lower and more volatile availability than CPU-based instances, making homogeneous clusters that depend on a single GPU type vulnerable to correlated failures. Heterogeneous clusters spanning multiple GPU types can address this by leveraging complementary availability patterns across diverse spot pools, yet existing LLM serving systems are designed for homogeneous environments and suffer from load imbalance when deployed on heterogeneous GPUs. This paper presents ShuntServe, a cost-efficient LLM serving system for heterogeneous spot GPU clusters. ShuntServe employs a roofline model-based analytical serving performance estimator and a dynamic programming-based model placement optimizer that jointly determines node configuration, parallelization strategy, and layer assignment to maximize throughput across heterogeneous GPUs. To enhance fault tolerance when using spot instances, ShuntServe combines output-preserving request migration with concurrent initialization via a shared tensor store, minimizing migration downtime by overlapping replacement node preparation with ongoing serving. Evaluation on Llama-3.1-70B and Qwen3-32B with a heterogeneous AWS cluster of L4, A10G, and L40S GPUs shows that ShuntServe achieves 1.42x and 1.35x higher throughput than state-of-the-art baselines and attains 31.9% and 31.2% cost efficiency improvements over on-demand instances for offline and online serving, respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 51 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

29 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  2. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  3. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
12 more
  1. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  3. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  4. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  5. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  6. 2021A Runtime-Based Computational Performance Predictor for Deep Neural Network Trainingcited by 29
  7. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 20
  8. 2025Demystifying NCCL: An In-depth Analysis of GPU Communication Protocols and Algorithmscited by 20
  9. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 17
  10. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  11. 2020Hierarchical Roofline Performance Analysis for Deep Learning Applicationscited by 11
  12. 2022SpotLake: Diverse Spot Instance Dataset Archive Servicecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
  2. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization
  3. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
  4. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  5. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
  6. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  7. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  8. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment