mlsanityData to 7 Oct 2026

Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs

arXiv 2502.00722 ↗First posted 2 Feb 2025cs.DCcited by 21

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However, this in turn degrades the cost-efficiency of LLM serving as common practices primarily rely on homogeneous GPU resources. In response to this problem, this work conducts a thorough study about serving LLMs over heterogeneous GPU resources on cloud platforms. The rationale is that different GPU types exhibit distinct compute and memory characteristics, aligning well with the divergent resource demands of diverse requests. Particularly, through comprehensive benchmarking, we discover that the cost-efficiency of LLM serving can be substantially optimized by meticulously determining GPU composition, deployment configurations, and workload assignments. Subsequently, we design a scheduling algorithm via mixed-integer linear programming, aiming at deducing the most cost-efficient serving plan under the constraints of price budget and real-time GPU availability. Remarkably, our approach effectively outperforms homogeneous and heterogeneous baselines under a wide array of scenarios, covering diverse workload traces, varying GPU availablilities, and multi-model serving. This casts new light on more accessible and efficient LLM serving over heterogeneous cloud resources.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 50 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

31 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Queue management for slo-oriented large language model serving5 Jun
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization2 Mar
  5. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  8. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2024Mixtral of Expertscited by 1,785
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  10. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  11. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
11 more
  1. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  2. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  3. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  4. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  5. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  6. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  7. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  8. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  9. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  10. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5
  11. 2025CuAsmRL: Optimizing GPU SASS Schedules via Deep Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 15.

21 papers.

Sort:

By area:

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  3. 2025Parallax: Efficient LLM Inference Service over Decentralized Environmentcited by 12
  4. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 8
  5. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  6. 2025Cascadia: An Efficient Cascade Serving System for Large Language Modelscited by 5
  7. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  8. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  9. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methodscited by 3
  10. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  11. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inferencecited by 2
  12. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
9 more
  1. 2026Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Servingcited by 1
  2. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 1
  3. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  4. 2026ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusterscited by 1
  5. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
  6. 2026Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Cloudscited by 0
  7. 2026ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inferencecited by 0
  8. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  9. 2026EMA: Elastic and Performance Transparent Memory Across GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  3. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  4. 2026Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
  5. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
  6. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
  7. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
  8. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management