# Fast State Restoration in LLM Serving with HCache

arXiv: https://arxiv.org/abs/2410.05004 · HTML: https://mlsanity.com/paper/2410.05004/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM](https://mlsanity.com/paper/2403.05527/) (2024)
- [DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference](https://mlsanity.com/paper/2401.08671/) (2024)
- [No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization](https://mlsanity.com/paper/2402.18096/) (2024)
- [KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization](https://mlsanity.com/paper/2405.03917/) (2024)
- [QAQ: Quality Adaptive Quantization for LLM KV Cache](https://mlsanity.com/paper/2403.04643/) (2024)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)

## Built on by (most cited first, 30 of 33)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](https://mlsanity.com/paper/2509.21009/) (2025)
- [Asynchronous LLM Function Calling](https://mlsanity.com/paper/2412.07017/) (2024)
- [TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications](https://mlsanity.com/paper/2510.18586/) (2025)
- [DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference](https://mlsanity.com/paper/2602.21548/) (2026)
- [Pie: A Programmable Serving System for Emerging LLM Applications](https://mlsanity.com/paper/2510.24051/) (2025)
- [KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation](https://mlsanity.com/paper/2411.17089/) (2024)
- [RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference](https://mlsanity.com/paper/2505.02922/) (2025)
- [TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling](https://mlsanity.com/paper/2510.02758/) (2025)
- [TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing](https://mlsanity.com/paper/2604.03143/) (2026)
- [OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration](https://mlsanity.com/paper/2601.10729/) (2026)
- [XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization](https://mlsanity.com/paper/2508.10395/) (2025)
- [KV Cache Translation across Heterogeneous Large Language Models](https://mlsanity.com/paper/2607.28979/) (2026)
- [MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection](https://mlsanity.com/paper/2506.19884/) (2025)
- [Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving](https://mlsanity.com/paper/2606.02964/) (2026)
- [SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer](https://mlsanity.com/paper/2607.28150/) (2026)
- [Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing](https://mlsanity.com/paper/2507.08045/) (2025)
- [Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling](https://mlsanity.com/paper/2507.18006/) (2025)
- [Efficient Remote KV Cache Reuse with GPU-native Video Codec](https://mlsanity.com/paper/2602.09725/) (2026)
- [CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration](https://mlsanity.com/paper/2604.25080/) (2026)
- [CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals](https://mlsanity.com/paper/2607.28263/) (2026)
- [Efficient Iterative Retrieval with Heterogeneous Batching](https://mlsanity.com/paper/2609.25405/) (2026)
- [HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement](https://mlsanity.com/paper/2510.20878/) (2025)
- [Serve Programs, Not Prompts](https://mlsanity.com/paper/2510.25412/) (2025)
- [When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving](https://mlsanity.com/paper/2605.04450/) (2026)
- [Bounded-State Restoration: Decoupling Local Restore Capacity from External LLM State](https://mlsanity.com/paper/2608.17826/) (2026)
- [AgentDecarbonizer: Carbon-Aware Execution for AI Agents](https://mlsanity.com/paper/2608.20566/) (2026)
- [mzCache: On-Device LLM Memory Management under Multitasking](https://mlsanity.com/paper/2609.01338/) (2026)
- [EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?](https://mlsanity.com/paper/2609.33762/) (2026)
- [TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash](https://mlsanity.com/paper/2609.35065/) (2026)

## Abstract (arXiv)

The growing complexity of LLM usage today, e.g., multi-round conversation and retrieval-augmented generation (RAG), makes contextual states (i.e., KV cache) reusable across user requests. Given the capacity constraints of GPU memory, only a limited number of contexts can be cached on GPU for reusing. Existing inference systems typically evict part of the KV cache and restore it by recomputing it from the original tokens or offloading it to host storage for later retrieval, both of which introduce substantial computational or I/O overheads. We propose HCache, a novel LLM state restoration method. Its key idea is to restore LLM states from intermediate activations and thus utilize computational and I/O resources with low overhead. We enhance HCache with two techniques, including i) a bubble-free restoration scheduler that integrates resource-complementary methods to optimize the balance between computation and IO tasks; and ii) a chunk-based storage manager to address the layout mismatch issue (i.e., layer-before-token saving versus token-before-layer restoration). Our evaluations, conducted using real-world tasks, show that HCache reduces the TTFT by up to 1.93X compared to KV offload while consuming 1.92-2.40X less storage space; compared to token recomputation, HCache achieves up to 5.73X reduction in TTFT.