# SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models

arXiv: https://arxiv.org/abs/2406.00025 · HTML: https://mlsanity.com/paper/2406.00025/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 16 of 16)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [NetLLM: Adapting Large Language Models for Networking](https://mlsanity.com/paper/2402.02338/) (2024)
- [DeepCache: Principled Cache for Mobile Deep Vision](https://mlsanity.com/paper/1712.01670/) (2017)
- [An Imitation Learning Approach for Cache Replacement](https://mlsanity.com/paper/2006.16239/) (2020)
- [On Optimal Caching and Model Multiplexing for Large Model Inference](https://mlsanity.com/paper/2306.02003/) (2023)
- [Sparks of GPTs in Edge Intelligence for Metaverse: Caching and Inference for Mobile AIGC Services](https://mlsanity.com/paper/2304.08782/) (2023)
- [Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models](https://mlsanity.com/paper/2310.13395/) (2023)

## Built on by (most cited first, 16 of 16)

- [vCache: Verified Semantic Prompt Caching](https://mlsanity.com/paper/2502.03771/) (2025)
- [Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation](https://mlsanity.com/paper/2508.07675/) (2025)
- [InstCache: A Predictive Cache for LLM Serving](https://mlsanity.com/paper/2411.13820/) (2024)
- [Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models](https://mlsanity.com/paper/2505.11271/) (2025)
- [SemanticALLI: Caching Reasoning, Not Just Responses, in Agentic Systems](https://mlsanity.com/paper/2601.16286/) (2026)
- [From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching](https://mlsanity.com/paper/2601.23088/) (2026)
- [Generative Caching for Structurally Similar Prompts and Responses](https://mlsanity.com/paper/2511.17565/) (2025)
- [TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents](https://mlsanity.com/paper/2602.10986/) (2026)
- [Asynchronous Verified Semantic Caching for Tiered LLM Architectures](https://mlsanity.com/paper/2602.13165/) (2026)
- [EACO-RAG: Towards Distributed Tiered LLM Deployment using Edge-Assisted and Collaborative RAG with Adaptive Knowledge Update](https://mlsanity.com/paper/2410.20299/) (2024)
- [An Ensemble Embedding Approach for Improving Semantic Caching Performance in LLM-based Systems](https://mlsanity.com/paper/2507.07061/) (2025)
- [AVEC: Bootstrapping Privacy for Local LLMs](https://mlsanity.com/paper/2509.10561/) (2025)
- [MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference](https://mlsanity.com/paper/2603.26557/) (2026)
- [MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation](https://mlsanity.com/paper/2605.24914/) (2026)
- [PLACEMEM: Toward a Compute-Aware Memory Plane for Lifelong Agents](https://mlsanity.com/paper/2607.04089/) (2026)
- [Similarity Is Not Validity: Defending LLM Semantic Caches Against Poisoning](https://mlsanity.com/paper/2609.35908/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become increasingly popular, transforming a wide range of applications across various domains. However, the real-world effectiveness of their query cache systems has not been thoroughly investigated. In this work, we for the first time conducted an analysis on real-world human-to-LLM interaction data, identifying key challenges in existing caching solutions for LLM-based chat services. Our findings reveal that current caching methods fail to leverage semantic connections, leading to inefficient cache performance and extra token costs. To address these issues, we propose SCALM, a new cache architecture that emphasizes semantic analysis and identifies significant cache entries and patterns. We also detail the implementations of the corresponding cache storage and eviction strategies. Our evaluations show that SCALM increases cache hit ratios and reduces operational costs for LLMChat services. Compared with other state-of-the-art solutions in GPTCache, SCALM shows, on average, a relative increase of 63% in cache hit ratio and a relative improvement of 77% in tokens savings.