mlsanityData to 7 Oct 2026

Fast State Restoration in LLM Serving with HCache

arXiv 2410.05004 ↗First posted 7 Oct 2024cs.DCcited by 32

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

The growing complexity of LLM usage today, e.g., multi-round conversation and retrieval-augmented generation (RAG), makes contextual states (i.e., KV cache) reusable across user requests. Given the capacity constraints of GPU memory, only a limited number of contexts can be cached on GPU for reusing. Existing inference systems typically evict part of the KV cache and restore it by recomputing it from the original tokens or offloading it to host storage for later retrieval, both of which introduce substantial computational or I/O overheads. We propose HCache, a novel LLM state restoration method. Its key idea is to restore LLM states from intermediate activations and thus utilize computational and I/O resources with low overhead. We enhance HCache with two techniques, including i) a bubble-free restoration scheduler that integrates resource-complementary methods to optimize the balance between computation and IO tasks; and ii) a chunk-based storage manager to address the layout mismatch issue (i.e., layer-before-token saving versus token-before-layer restoration). Our evaluations, conducted using real-world tasks, show that HCache reduces the TTFT by up to 1.93X compared to KV offload while consuming 1.92-2.40X less storage space; compared to token recomputation, HCache achieves up to 5.73X reduction in TTFT.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 62 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

27 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
14 more
  1. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  3. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  4. 2023Using an LLM to Help With Code Understandingcited by 222
  5. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  6. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  8. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  10. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  11. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  12. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  13. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  14. 2023Stateful Large Language Model Serving with Pensievecited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 18.

32 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 31
  3. 2024Asynchronous LLM Function Callingcited by 18
  4. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  5. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
  6. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  7. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  8. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  9. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  10. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  11. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
  12. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
20 more
  1. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  2. 2025MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selectioncited by 2
  3. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
  4. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  5. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharingcited by 1
  6. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scalingcited by 1
  7. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  8. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  9. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  10. 2025Serve Programs, Not Promptscited by 0
  11. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 0
  12. 2026When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Servingcited by 0
  13. 2026Bounded-State Restoration: Decoupling Local Restore Capacity from External LLM Statecited by 0
  14. 2026AgentDecarbonizer: Carbon-Aware Execution for AI Agentscited by 0
  15. 2026mzCache: On-Device LLM Memory Management under Multitaskingcited by 0
  16. 2026Efficient Iterative Retrieval with Heterogeneous Batchingcited by 0
  17. 2026EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?cited by 0
  18. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flashcited by 0
  19. 2026Persistent Context Graphs for Efficient Memory Compaction in LLM Agentscited by 0
  20. 2026Cache the Encoder Within:Compact, Reusable Memory across LLM Queriescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  2. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  3. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Serving
  4. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  5. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving
  8. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs