mlsanityData to 7 Oct 2026

Stateful Large Language Model Serving with Pensieve

arXiv 2312.05516 ↗First posted 9 Dec 2023cs.LG cs.DCcited by 59

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) are wildly popular today and it is important to serve them efficiently. Existing LLM serving systems are stateless across requests. Consequently, when LLMs are used in the common setting of multi-turn conversations, a growing log of the conversation history must be processed alongside any request by the serving system at each turn, resulting in repeated processing. In this paper, we design Pensieve, a system optimized for multi-turn conversation LLM serving. Pensieve maintains the conversation state across requests by caching previously processed history to avoid duplicate processing. Pensieve's multi-tier caching strategy can utilize both GPU and CPU memory to efficiently store and retrieve cached data. Pensieve also generalizes the recent PagedAttention kernel to support attention between multiple input tokens with a GPU cache spread over non-contiguous memory. Our evaluation shows that Pensieve can achieve 1.14-3.0× the throughput of vLLM and TensorRT-LLM and significantly reduce latency.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 60 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

29 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2017Mixed Precision Trainingcited by 1,135
15 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2021A White Paper on Neural Network Quantizationcited by 453
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  9. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  11. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  12. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  13. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  14. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  15. 2021Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecturecited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 22; 2026: 28.

59 papers.

Sort:

By area:

  1. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  7. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  8. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  9. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  10. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  11. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  12. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
38 more
  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  2. 2024Asynchronous LLM Function Callingcited by 18
  3. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  4. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  5. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  6. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
  7. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  8. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  9. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  10. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
  11. 2026Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Servingcited by 7
  12. 2025Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoringcited by 6
  13. 2026You Need an Encoder for Native Position-Independent Cachingcited by 6
  14. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  15. 2025Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Managementcited by 4
  16. 2026Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMscited by 4
  17. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  18. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agentscited by 3
  19. 2025Generative AI for CAD Automation: Leveraging Large Language Models for 3D Modellingcited by 2
  20. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  21. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  22. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
  23. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharingcited by 2
  24. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  25. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  26. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  27. 2025EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reusecited by 1
  28. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharingcited by 1
  29. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systemscited by 1
  30. 2026A CXL Memory Rack for Multi-Turn LLM Servingcited by 1
  31. 2025Network and Systems Performance Characterization of MCP-Enabled LLM Agentscited by 0
  32. 2025DisCEdge: Distributed Context Management for Large Language Models at the Edgecited by 0
  33. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
  34. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Servicecited by 0
  35. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  36. 2026Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networkscited by 0
  37. 2026Not Forgotten: Implementation and Evaluation of a Personalized Episodic Memory for the Humanoid Robot Head Kimcited by 0
  38. 2026ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platformcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  2. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
  3. 2025IC-Cache: Efficient Large Language Model Serving via In-context Caching
  4. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  5. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management
  6. 2025Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
  7. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
  8. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics