# Auditing Prompt Caching in Language Model APIs

arXiv: https://arxiv.org/abs/2502.07776 · HTML: https://mlsanity.com/paper/2502.07776/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition](https://mlsanity.com/paper/2402.15220/) (2024)
- [Hydragen: High-Throughput LLM Inference with Shared Prefixes](https://mlsanity.com/paper/2402.05099/) (2024)
- [Spectre Attacks: Exploiting Speculative Execution](https://mlsanity.com/paper/1801.01203/) (2018)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems](https://mlsanity.com/paper/2409.20002/) (2024)
- [What Was Your Prompt? A Remote Keylogging Attack on AI Assistants](https://mlsanity.com/paper/2403.09751/) (2024)
- [InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks](https://mlsanity.com/paper/2411.18191/) (2024)
- [Remote Timing Attacks on Efficient Language Model Inference](https://mlsanity.com/paper/2410.17175/) (2024)
- [When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs](https://mlsanity.com/paper/2411.01076/) (2024)

## Built on by (most cited first, 20 of 20)

- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2601.06007/) (2026)
- [Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test](https://mlsanity.com/paper/2506.06975/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching](https://mlsanity.com/paper/2601.23088/) (2026)
- [TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents](https://mlsanity.com/paper/2602.10986/) (2026)
- [HijackKV: New Threat in Position-Independent KV Cache Reuse](https://mlsanity.com/paper/2607.19957/) (2026)
- [AttestLLM: Efficient Attestation Framework for Billion-scale On-device LLMs](https://mlsanity.com/paper/2509.06326/) (2025)
- [Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries](https://mlsanity.com/paper/2509.22202/) (2025)
- [OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services](https://mlsanity.com/paper/2602.20595/) (2026)
- [PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems](https://mlsanity.com/paper/2603.10726/) (2026)
- [Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways](https://mlsanity.com/paper/2604.21083/) (2026)
- [VOW: Verifiable and Oblivious Watermark Detection for Large Language Models](https://mlsanity.com/paper/2604.27666/) (2026)
- [CacheProbe: Auditing Prompt Cache Isolation in Gateway APIs](https://mlsanity.com/paper/2605.30613/) (2026)
- [Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching](https://mlsanity.com/paper/2607.15516/) (2026)
- [LaCache: Robust Semantic Caching for LLM Serving](https://mlsanity.com/paper/2608.01718/) (2026)
- [Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served](https://mlsanity.com/paper/2603.02348/) (2026)
- [Shape and Substance: Dual-Layer Side-Channel Attacks on Local Vision-Language Models](https://mlsanity.com/paper/2603.25403/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing](https://mlsanity.com/paper/2607.20723/) (2026)

## Abstract (arXiv)

Prompt caching in large language models (LLMs) results in data-dependent timing variations: cached prompts are processed faster than non-cached prompts. These timing differences introduce the risk of side-channel timing attacks. For example, if the cache is shared across users, an attacker could identify cached prompts from fast API response times to learn information about other users' prompts. Because prompt caching may cause privacy leakage, transparency around the caching policies of API providers is important. To this end, we develop and conduct statistical audits to detect prompt caching in real-world LLM API providers. We detect global cache sharing across users in seven API providers, including OpenAI, resulting in potential privacy leakage about users' prompts. Timing variations due to prompt caching can also result in leakage of information about model architecture. Namely, we find evidence that OpenAI's embedding model is a decoder-only Transformer, which was previously not publicly known.