# MeanCache: User-Centric Semantic Caching for LLM Web Services

arXiv: https://arxiv.org/abs/2403.02694 · HTML: https://mlsanity.com/paper/2403.02694/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 55 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [How to Train Data-Efficient LLMs](https://mlsanity.com/paper/2402.09668/) (2024)
- [Compositional Generative Modeling: A Single Model is Not All You Need](https://mlsanity.com/paper/2402.01103/) (2024)
- [FedDebug: Systematic Debugging for Federated Learning Applications](https://mlsanity.com/paper/2301.03553/) (2023)
- [FedDefender: Backdoor Attack Defense in Federated Learning](https://mlsanity.com/paper/2307.08672/) (2023)
- [Towards cost-effective and resource-aware aggregation at Edge for Federated Learning](https://mlsanity.com/paper/2204.07767/) (2022)
- [TraceFL: Interpretability-Driven Debugging in Federated Learning via Neuron Provenance](https://mlsanity.com/paper/2312.13632/) (2023)

## Built on by (most cited first, 14 of 14)

- [Towards AI Search Paradigm](https://mlsanity.com/paper/2506.17188/) (2025)
- [Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data](https://mlsanity.com/paper/2504.02268/) (2025)
- [Category-Aware Semantic Caching for Heterogeneous LLM Workloads](https://mlsanity.com/paper/2510.26835/) (2025)
- [From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching](https://mlsanity.com/paper/2601.23088/) (2026)
- [Asynchronous Verified Semantic Caching for Tiered LLM Architectures](https://mlsanity.com/paper/2602.13165/) (2026)
- [EACO-RAG: Towards Distributed Tiered LLM Deployment using Edge-Assisted and Collaborative RAG with Adaptive Knowledge Update](https://mlsanity.com/paper/2410.20299/) (2024)
- [From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings](https://mlsanity.com/paper/2603.03301/) (2026)
- [MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference](https://mlsanity.com/paper/2603.26557/) (2026)
- [Closing the Operational Gap in Semantic Caching](https://mlsanity.com/paper/2606.19719/) (2026)
- [Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systems](https://mlsanity.com/paper/2601.11687/) (2026)
- [Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey](https://mlsanity.com/paper/2604.07857/) (2026)
- [Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders](https://mlsanity.com/paper/2608.20280/) (2026)
- [On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation](https://mlsanity.com/paper/2609.33141/) (2026)
- [Similarity Is Not Validity: Defending LLM Semantic Caches Against Poisoning](https://mlsanity.com/paper/2609.35908/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) like ChatGPT and Llama have revolutionized natural language processing and search engine dynamics. However, these models incur exceptionally high computational costs. For instance, GPT-3 consists of 175 billion parameters, where inference demands billions of floating-point operations. Caching is a natural solution to reduce LLM inference costs on repeated queries, which constitute about 31% of the total queries. However, existing caching methods are incapable of finding semantic similarities among LLM queries nor do they operate on contextual queries, leading to unacceptable false hit-and-miss rates. This paper introduces MeanCache, a user-centric semantic cache for LLM-based services that identifies semantically similar queries to determine cache hit or miss. Using MeanCache, the response to a user's semantically similar query can be retrieved from a local cache rather than re-querying the LLM, thus reducing costs, service provider load, and environmental impact. MeanCache leverages Federated Learning (FL) to collaboratively train a query similarity model without violating user privacy. By placing a local cache in each user's device and using FL, MeanCache reduces the latency and costs and enhances model performance, resulting in lower false hit rates. MeanCache also encodes context chains for every cached query, offering a simple yet highly effective mechanism to discern contextual query responses from standalone. Our experiments benchmarked against the state-of-the-art caching method, reveal that MeanCache attains an approximately 17% higher F-score and a 20% increase in precision during semantic cache hit-and-miss decisions while performing even better on contextual queries. It also reduces the storage requirement by 83% and accelerates semantic cache hit-and-miss decisions by 11%.