mlsanityData to 14 Oct 2026

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

arXiv 2311.04934 ↗First posted 7 Nov 2023cs.CL cs.AIcited by 268

Not on the map. Why?

Cited by 126 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have overlapping text segments, such as system messages, prompt templates, and documents provided for context. Our key insight is that by precomputing and storing the attention states of these frequently occurring text segments on the inference server, we can efficiently reuse them when these segments appear in user prompts. Prompt Cache employs a schema to explicitly define such reusable text segments, called prompt modules. The schema ensures positional accuracy during attention state reuse and provides users with an interface to access cached states in their prompt. Using a prototype implementation, we evaluate Prompt Cache across several LLMs. We show that Prompt Cache significantly reduce latency in time-to-first-token, especially for longer prompts such as document-based question answering and recommendations. The improvements range from 8x for GPU-based inference to 60x for CPU-based inference, all while maintaining output accuracy and without the need for model parameter modifications.

Key facts. Cited by 268 papers in the map (data through 14 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

34 of 45 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
22 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  8. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2023A Survey on Large Language Models for Recommendationcited by 379
  13. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  14. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  15. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  16. 2021Efficient Attentions for Long Document Summarizationcited by 345
  17. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 112
  18. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  19. 2021Position Information in Transformers: An Overviewcited by 103
  20. 2020Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Datacited by 87
  21. 2021MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Educationcited by 37
  22. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 40; 2025: 69; 2026: 153.

268 papers.

Sort:

By area:

  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  3. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  4. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  6. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  7. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  8. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 109
  9. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  10. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  12. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
38 more
  1. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  2. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  3. 2023Stateful Large Language Model Serving with Pensievecited by 61
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  5. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 58
  6. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  7. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
  8. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  9. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  10. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
  11. 2024Block-Attention for Efficient Prefillingcited by 37
  12. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  13. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  14. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  15. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  16. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  17. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  18. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  19. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  20. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 27
  21. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 26
  22. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  23. 2024SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactionscited by 25
  24. 2023TypeFly: Flying Drones with Large Language Modelcited by 23
  25. 2024Optimizing LLM Queries in Relational Data Analytics Workloadscited by 22
  26. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  27. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning taskscited by 22
  28. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
  29. 2025Auditing Prompt Caching in Language Model APIscited by 20
  30. 2025COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Contextcited by 20
  31. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  32. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 19
  33. 2025KVComm: Enabling Efficient LLM Communication through Selective KV Sharingcited by 18
  34. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  35. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
  36. 2024Do Large Language Models Need a Content Delivery Network?cited by 16
  37. 2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 16
  38. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16

50 most cited of 268 All 268 (JSON)