mlsanityData to 14 Oct 2026

Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading

arXiv 2504.11816 ↗First posted 16 Apr 2025cs.LG cs.DCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 22 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

13 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  2. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving18 Oct
  3. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving11 May

Other references · 10

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2022Efficiently Scaling Transformer Inferencecited by 539
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 31
  8. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  9. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  10. 2024DeepVM: Integrating Spot and On-Demand VMs for Cost-Efficient Deep Learning Clusters in the Cloudcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  2. 2026ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  2. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
  5. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
  6. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  7. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  8. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference