mlsanityData to 14 Oct 2026

Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

arXiv 2508.18736 ↗First posted 26 Aug 2025cs.DB cs.LGcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Serving Large Language Models (LLMs) at scale requires meeting strict Service Level Objectives (SLOs) under severe computational and memory constraints. Nevertheless, traditional caching strategies fall short: exact-matching and prefix caches neglect query semantics, while state-of-the-art semantic caches remain confined to traditional intuitions, offering little conceptual departure. Building on this, we present SISO, a semantic caching system that redefines efficiency for LLM serving. SISO introduces centroid-based caching to maximize coverage with minimal memory, locality-aware replacement to preserve high-value entries, and dynamic thresholding to balance accuracy and latency under varying workloads. Across diverse datasets, SISO delivers up to 1.71× higher hit ratios and consistently stronger SLO attainment compared to state-of-the-art systems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 31 of its 69 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

31 of 69 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
19 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  8. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  9. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  10. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  11. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  12. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  13. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 52
  14. 2020Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQscited by 48
  15. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  16. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 24
  17. 2017Semantic Structure and Interpretability of Word Embeddingscited by 21
  18. 2024Aladdin: Joint Placement and Scaling for SLO-Aware LLM Servingcited by 11
  19. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Cachingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoderscited by 0
  2. 2026PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
  2. 2025IC-Cache: Efficient Large Language Model Serving via In-context Caching
  3. 2026Asynchronous Verified Semantic Caching for Tiered LLM Architectures
  4. 2026Continuous Semantic Caching for Low-Cost LLM Serving
  5. 2026Kalypso: Relational LLM Serving
  6. 2025Category-Aware Semantic Caching for Heterogeneous LLM Workloads
  7. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  8. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse