mlsanityData to 7 Oct 2026

Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service

arXiv 2603.08739 ↗First posted 25 Feb 2026cs.AR cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity motivates offloading the KV cache to cheaper external storage tiers. While this expands capacity, it introduces the challenge of dynamically managing heterogeneous storage resources to balance cost, throughput, and latency under varying workloads. We formulate this as a multi-objective optimization problem: identifying the Pareto frontier across these metrics within the storage configuration space. Using a high-fidelity end-to-end simulator, we observe that the objective functions are non-analytic and exhibit complex variable coupling, making the Pareto frontier difficult to approximate analytically. To obtain the frontier, we introduce Kareto, a KV-cache Adaptive REsource managemenT Optimizer. Kareto leverages a diminishing-return-guided pruning method to efficiently navigate the large configuration space and approximate the Pareto frontier. Additionally, it incorporates a fine-grained adaptive tuner that uses eviction policies in tier storage and KV block access patterns for group-specific cache management, improving cache efficiency. Experiments on real-world traces show that Kareto adapts to workload and can identify configurations of better cost efficiency, covering static strategies. Compared to the fixed setup with 1024 GB DRAM, Kareto can improve throughput by up to 9.3%, or reduce latency by up to 58.3%, or lower cost by up to 20.2% under respective optimization objectives.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 29 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

17 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider3 Jun
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  4. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  10. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  11. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  12. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  4. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  5. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  7. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  8. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference