# vCache: Verified Semantic Prompt Caching

arXiv: https://arxiv.org/abs/2502.03771 · HTML: https://mlsanity.com/paper/2502.03771/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [The Extreme Value Machine](https://mlsanity.com/paper/1506.06112/) (2015)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [Survey of Vector Database Management Systems](https://mlsanity.com/paper/2310.14021/) (2023)
- [ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search](https://mlsanity.com/paper/2006.05324/) (2020)
- [Large Search Model: Redefining Search Stack in the Era of LLMs](https://mlsanity.com/paper/2310.14587/) (2023)
- [SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models](https://mlsanity.com/paper/2406.00025/) (2024)
- [Efficient Prompt Caching via Embedding Similarity](https://mlsanity.com/paper/2402.01173/) (2024)
- [Threshold-Consistent Margin Loss for Open-World Deep Metric Learning](https://mlsanity.com/paper/2307.04047/) (2023)
- [SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models](https://mlsanity.com/paper/2210.04802/) (2022)

## Built on by (most cited first, 17 of 17)

- [From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching](https://mlsanity.com/paper/2601.23088/) (2026)
- [TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents](https://mlsanity.com/paper/2602.10986/) (2026)
- [Asynchronous Verified Semantic Caching for Tiered LLM Architectures](https://mlsanity.com/paper/2602.13165/) (2026)
- [Supporting Dynamic Agentic Workloads: How Data and Agents Interact](https://mlsanity.com/paper/2512.09548/) (2025)
- [QVCache: A Query-Aware Vector Cache](https://mlsanity.com/paper/2602.02057/) (2026)
- [StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving](https://mlsanity.com/paper/2603.28795/) (2026)
- [From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings](https://mlsanity.com/paper/2603.03301/) (2026)
- [Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Betting](https://mlsanity.com/paper/2603.08907/) (2026)
- [LaCache: Robust Semantic Caching for LLM Serving](https://mlsanity.com/paper/2608.01718/) (2026)
- [Why Agent Caching Fails and How to Fix It: Structured Intent Canonicalization with Few-Shot Learning](https://mlsanity.com/paper/2602.18922/) (2026)
- [MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation](https://mlsanity.com/paper/2605.24914/) (2026)
- [Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders](https://mlsanity.com/paper/2608.20280/) (2026)
- [PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving](https://mlsanity.com/paper/2609.10372/) (2026)
- [On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation](https://mlsanity.com/paper/2609.33141/) (2026)
- [Similarity Is Not Validity: Defending LLM Semantic Caches Against Poisoning](https://mlsanity.com/paper/2609.35908/) (2026)
- [Bellman-Centric Learning: Near-Optimal Regret for Linear Bandits with Memory](https://mlsanity.com/paper/2610.05659/) (2026)
- [Semantic Views: Certified Reuse of LLM Labels Across Semantic Queries](https://mlsanity.com/paper/2610.12768/) (2026)

## Abstract (arXiv)

Semantic caches return cached responses for semantically similar prompts to reduce LLM inference latency and cost. They embed cached prompts and store them alongside their response in a vector database. Embedding similarity metrics assign a numerical score to quantify the similarity between a request and its nearest neighbor prompt from the cache. Existing systems use the same static similarity threshold across all requests to determine whether two prompts can share similar responses. However, we observe that static thresholds do not give formal correctness guarantees, result in unexpected error rates, and lead to suboptimal cache hit rates. This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees for predictable performance. It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training. Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines with up to 12.5× higher cache hit and 26× lower error rates. We release the vCache implementation and four benchmarks to support future research.