mlsanityData to 14 Oct 2026

Category-Aware Semantic Caching for Heterogeneous LLM Workloads

arXiv 2510.26835 ↗First posted 29 Oct 2025cs.DB cs.AI cs.LGcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLM serving systems process heterogeneous query workloads where different categories exhibit different characteristics. Code queries cluster densely in embedding space while conversational queries distribute sparsely. Content staleness varies from minutes (stock data) to months (code patterns). Query repetition patterns range from power-law (code) to uniform (conversation), producing long tail cache hit rate distributions: high-repetition categories achieve 40-60% hit rates while low-repetition or volatile categories achieve 5-15% hit rates. Vector databases must exclude the long tail because remote search costs (30ms) require 15--20% hit rates to break even, leaving 20-30% of production traffic uncached. Uniform cache policies compound this problem: fixed thresholds cause false positives in dense spaces and miss valid paraphrases in sparse spaces; fixed TTLs waste memory or serve stale data. This paper presents category-aware semantic caching where similarity thresholds, TTLs, and quotas vary by query category. We present a hybrid architecture separating in-memory HNSW search from external document storage, reducing miss cost from 30ms to 2ms. This reduction makes low-hit-rate categories economically viable (break-even at 3-5% versus 15-20%), enabling cache coverage across the entire workload distribution. Adaptive load-based policies extend this framework to respond to downstream model load, dynamically adjusting thresholds and TTLs to reduce traffic to overloaded models by 9-17% in theoretical projections.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 8 of its 21 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

8 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse17 Mar

Other references · 7

  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  3. 2017The Case for Learned Index Structurescited by 188
  4. 2023Survey of Vector Database Management Systemscited by 39
  5. 2024MeanCache: User-Centric Semantic Caching for LLM Web Servicescited by 14
  6. 2024LLMs for Test Input Generation for Semantic Cachescited by 4
  7. 2025Toward Understanding Bugs in Vector Database Management Systemscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMscited by 2
  2. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  3. 2026MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inferencecited by 1
  4. 2026Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
  2. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
  3. 2026Asynchronous Verified Semantic Caching for Tiered LLM Architectures
  4. 2026Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
  5. 2026Continuous Semantic Caching for Low-Cost LLM Serving
  6. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  7. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  8. 2026Kalypso: Relational LLM Serving