# From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

arXiv: https://arxiv.org/abs/2603.03301 · HTML: https://mlsanity.com/paper/2603.03301/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.12457/) (2024)
- [GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching](https://mlsanity.com/paper/2411.05276/) (2024)
- [vCache: Verified Semantic Prompt Caching](https://mlsanity.com/paper/2502.03771/) (2025)
- [MeanCache: User-Centric Semantic Caching for LLM Web Services](https://mlsanity.com/paper/2403.02694/) (2024)
- [InstCache: A Predictive Cache for LLM Serving](https://mlsanity.com/paper/2411.13820/) (2024)
- [Embedding Recycling for Language Models](https://mlsanity.com/paper/2207.04993/) (2022)
- [A Generative Caching System for Large Language Models](https://mlsanity.com/paper/2503.17603/) (2025)

## Built on by (most cited first, 1 of 1)

- [Closing the Operational Gap in Semantic Caching](https://mlsanity.com/paper/2606.19719/) (2026)

## Abstract (arXiv)

The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.