mlsanityData to 7 Oct 2026

ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

arXiv 2502.09334 ↗First posted 13 Feb 2025cs.DCcited by 23

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Recent developments in large language models (LLMs) have demonstrated their remarkable proficiency in a range of tasks. Compared to in-house homogeneous GPU clusters, deploying LLMs in cloud environments with diverse types of GPUs is crucial for addressing the GPU shortage problem and being more cost-effective. However, the diversity of network environments and various GPU types on the cloud bring difficulties to achieving high-performance serving. In this work, we propose ThunderServe, a high-performance and cost-efficient LLM serving system for heterogeneous cloud environments. We introduce a novel scheduling algorithm, which optimizes the deployment plan of LLM serving to accommodate the heterogeneous resource and network bandwidth conditions in cloud environments. Furthermore, we propose a lightweight re-scheduling mechanism, designed to adapt to fluctuating online conditions (e.g., node failures, workload shifts) without the need for costly restarts of ongoing services. Empirical results in both heterogeneous cloud and homogeneous in-house environments reveal that ThunderServe delivers up to a 2.1× and on average a 1.7× increase in throughput and achieves up to a 2.5× and on average a 1.5× reduction in latency deadlines compared with state-of-the-art systems given the same price budget, suggesting opting for cloud services provides a more cost-efficient solution.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 43 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

19 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  2. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  3. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  10. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  11. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  12. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
4 more
  1. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 16
  2. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 13
  3. 2021Model-Parallel Model Selection for Deep Learning Systemscited by 6
  4. 2023How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Studycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  2. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMindcited by 10
  3. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  4. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  5. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  6. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  7. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  8. 2026Towards Resiliency in Large Language Model Serving with KevlarFlowcited by 3
  9. 2025Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloadingcited by 2
  10. 202510Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Trainingcited by 2
  11. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  12. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
11 more
  1. 2025Cost-Aware Model Orchestration for LLM-based Systemscited by 1
  2. 2026Quantization-Aware Collaborative Inference for Large Embodied AI Modelscited by 1
  3. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  4. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  5. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  6. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusterscited by 1
  7. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  8. 2026Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregationcited by 0
  9. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environmentscited by 0
  10. 2026OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Servingcited by 0
  11. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
  2. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
  3. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  4. 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
  5. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  6. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  8. 2025MoLink: Distributed and Efficient Serving Framework for Large Models