# An Ensemble Embedding Approach for Improving Semantic Caching Performance in LLM-based Systems

arXiv: https://arxiv.org/abs/2507.07061 · HTML: https://mlsanity.com/paper/2507.07061/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models](https://mlsanity.com/paper/2406.00025/) (2024)
- [Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity](https://mlsanity.com/paper/1911.03700/) (2019)
- [Word Embedding for Social Sciences: An Interdisciplinary Survey](https://mlsanity.com/paper/2207.03086/) (2022)
- [A Generative Caching System for Large Language Models](https://mlsanity.com/paper/2503.17603/) (2025)
- [Improving information retrieval through correspondence analysis instead of latent semantic analysis](https://mlsanity.com/paper/2303.08030/) (2023)

## Built on by (most cited first, 1 of 1)

- [SemanticALLI: Caching Reasoning, Not Just Responses, in Agentic Systems](https://mlsanity.com/paper/2601.16286/) (2026)

## Abstract (arXiv)

Semantic caching enhances the efficiency of large language model (LLM) systems by identifying semantically similar queries, storing responses once, and serving them for subsequent equivalent requests. However, existing semantic caching frameworks rely on single embedding models for query representation, which limits their ability to capture the diverse semantic relationships present in real-world query distributions. This paper presents an ensemble embedding approach that combines multiple embedding models through a trained meta-encoder to improve semantic similarity detection in LLM caching systems. We evaluate our method using the Quora Question Pairs (QQP) dataset, measuring cache hit ratios, cache miss ratios, token savings, and response times. Our ensemble approach achieves a 92% cache hit ratio for semantically equivalent queries while maintaining an 85% accuracy in correctly rejecting non-equivalent queries as cache misses. These results demonstrate that ensemble embedding methods significantly outperform single-model approaches in distinguishing between semantically similar and dissimilar queries, leading to more effective caching performance and reduced computational overhead in LLM-based systems.