# MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation

arXiv: https://arxiv.org/abs/2605.24914 · HTML: https://mlsanity.com/paper/2605.24914/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 24 of 24)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [PLAID: An Efficient Engine for Late Interaction Retrieval](https://mlsanity.com/paper/2205.09707/) (2022)
- [RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark](https://mlsanity.com/paper/2306.17100/) (2023)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [Query expansion techniques for information retrieval: A survey](https://mlsanity.com/paper/1708.00247/) (2017)
- [CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval](https://mlsanity.com/paper/2211.10411/) (2022)
- [vCache: Verified Semantic Prompt Caching](https://mlsanity.com/paper/2502.03771/) (2025)
- [SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models](https://mlsanity.com/paper/2406.00025/) (2024)
- [Deep Reinforced Query Reformulation for Information Retrieval](https://mlsanity.com/paper/2007.07987/) (2020)
- [Efficient Prompt Caching via Embedding Similarity](https://mlsanity.com/paper/2402.01173/) (2024)
- [Threshold-Consistent Margin Loss for Open-World Deep Metric Learning](https://mlsanity.com/paper/2307.04047/) (2023)
- [SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models](https://mlsanity.com/paper/2210.04802/) (2022)
- [POQD: Performance-Oriented Query Decomposer for Multi-vector retrieval](https://mlsanity.com/paper/2505.19189/) (2025)
- [Generalized Pseudo-Relevance Feedback](https://mlsanity.com/paper/2510.25488/) (2025)

## Abstract (arXiv)

To reduce LLM costs and latency, semantic caching systems must accurately identify when a new prompt matches a cached one. Current methods often rely on simplistic similarity measures, which limit their effectiveness. We introduce MVR-cache, a novel semantic caching approach that significantly improves retrieval accuracy by integrating Multi-Vector Retrieval (MVR). MVR-cache is built upon a learnable segmentation model that intelligently splits prompts, enabling fine-grained similarity comparisons via MaxSim. We derive the model's training objective from a rigorous theoretical analysis. This can ensure that optimizing this objective directly maximizes cache hits under strict correctness constraints. To solve the resulting non-differentiable combinatorial optimization problem, we leverage a reinforcement learning-based training strategy with the theoretically grounded objectives as the reward. Experimental results on established benchmarks across diverse tasks confirm that in comparison to the state-of-the-art, MVR-cache consistently increases the cache hit rates by up to 37% while maintaining the same correctness guarantees. MVR-cache is available at https://github.com/PKU-SDS-lab/MVR-Cache