# Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

arXiv: https://arxiv.org/abs/2607.15516 · HTML: https://mlsanity.com/paper/2607.15516/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 18 of 18)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [A Unified Approach to Routing and Cascading for LLMs](https://mlsanity.com/paper/2410.10347/) (2024)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [Budget-Aware Tool Use Enables Effective Agent Scaling](https://mlsanity.com/paper/2511.17006/) (2025)
- [Auditing Prompt Caching in Language Model APIs](https://mlsanity.com/paper/2502.07776/) (2025)
- [Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models](https://mlsanity.com/paper/2407.15504/) (2024)
- [Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2601.06007/) (2026)
- [xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning](https://mlsanity.com/paper/2510.08439/) (2025)
- [Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference](https://mlsanity.com/paper/2604.02985/) (2026)
- [Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Use](https://mlsanity.com/paper/2602.11541/) (2026)
- [ProCut: LLM Prompt Compression via Attribution Estimation](https://mlsanity.com/paper/2508.02053/) (2025)
- [Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2511.02130/) (2025)
- [Generative Caching for Structurally Similar Prompts and Responses](https://mlsanity.com/paper/2511.17565/) (2025)
- [CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflows](https://mlsanity.com/paper/2510.18043/) (2025)

## Built on by (most cited first, 1 of 1)

- [Protocol Compression Changes Which Party Pays: Bilateral Cost in Cross-Organization LLM Agent Communication](https://mlsanity.com/paper/2609.06129/) (2026)

## Abstract (arXiv)

Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios (r>=6). We propose Cache-Aware Prompt Compression (CAPC), pairing query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix (51.7% cost reduction at r=3); a graphify knowledge-graph RAG pipeline across two codebases (9.3x vs cache-all on FastAPI, 2.4x on httpx); and the public tau-bench retail benchmark (50 tasks), where CAPC is the cheapest of four strategies with reward exactly equal to vanilla (both 36/50, p=1.00) while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.