# GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching

arXiv: https://arxiv.org/abs/2411.05276 · HTML: https://mlsanity.com/paper/2411.05276/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 8 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 2 of 2)

- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)

## Built on by (most cited first, 22 of 22)

- [What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text](https://mlsanity.com/paper/2503.04804/) (2025)
- [SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching](https://mlsanity.com/paper/2509.24832/) (2025)
- [From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching](https://mlsanity.com/paper/2601.23088/) (2026)
- [Advancing Generative Artificial Intelligence and Large Language Models for Demand Side Management with Internet of Electric Vehicles](https://mlsanity.com/paper/2501.15544/) (2025)
- [Supporting Dynamic Agentic Workloads: How Data and Agents Interact](https://mlsanity.com/paper/2512.09548/) (2025)
- [From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings](https://mlsanity.com/paper/2603.03301/) (2026)
- [Beyond Linear LLM Invocation: An Efficient and Effective Semantic Filter Paradigm](https://mlsanity.com/paper/2603.04799/) (2026)
- [MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference](https://mlsanity.com/paper/2603.26557/) (2026)
- [Closing the Operational Gap in Semantic Caching](https://mlsanity.com/paper/2606.19719/) (2026)
- [LaCache: Robust Semantic Caching for LLM Serving](https://mlsanity.com/paper/2608.01718/) (2026)
- [Software Performance Engineering for Foundation Model-Powered Software](https://mlsanity.com/paper/2411.09580/) (2024)
- [Beyond the model: Key differentiators in large language models and multi-agent services](https://mlsanity.com/paper/2505.02489/) (2025)
- [ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling](https://mlsanity.com/paper/2507.08877/) (2025)
- [TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses](https://mlsanity.com/paper/2507.23674/) (2025)
- [MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems](https://mlsanity.com/paper/2508.21307/) (2025)
- [Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration](https://mlsanity.com/paper/2510.22679/) (2025)
- [Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systems](https://mlsanity.com/paper/2601.11687/) (2026)
- [Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching](https://mlsanity.com/paper/2602.22812/) (2026)
- [From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving](https://mlsanity.com/paper/2607.02574/) (2026)
- [DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving](https://mlsanity.com/paper/2609.31047/) (2026)
- [Similarity Is Not Validity: Defending LLM Semantic Caches Against Poisoning](https://mlsanity.com/paper/2609.35908/) (2026)
- [Semantic Views: Certified Reuse of LLM Labels Across Semantic Queries](https://mlsanity.com/paper/2610.12768/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs), such as GPT, have revolutionized artificial intelligence by enabling nuanced understanding and generation of human-like text across a wide range of applications. However, the high computational and financial costs associated with frequent API calls to these models present a substantial bottleneck, especially for applications like customer service chatbots that handle repetitive queries. In this paper, we introduce GPT Semantic Cache, a method that leverages semantic caching of query embeddings in in-memory storage (Redis). By storing embeddings of user queries, our approach efficiently identifies semantically similar questions, allowing for the retrieval of pre-generated responses without redundant API calls to the LLM. This technique achieves a notable reduction in operational costs while significantly enhancing response times, making it a robust solution for optimizing LLM-powered applications. Our experiments demonstrate that GPT Semantic Cache reduces API calls by up to 68.8% across various query categories, with cache hit rates ranging from 61.6% to 68.8%. Additionally, the system achieves high accuracy, with positive hit rates exceeding 97%, confirming the reliability of cached responses. This technique not only reduces operational costs, but also improves response times, enhancing the efficiency of LLM-powered applications.