mlsanityData to 7 Oct 2026

ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management

arXiv 2601.13631 ↗First posted 20 Jan 2026cs.OS cs.DCcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficiently serving Large Language Models (LLMs) with persistent Prefix Key-Value (KV) Cache is critical for applications like conversational search and multi-turn dialogue. Serving a request requires loading the pre-computed prefix KV cache and generating the first token, defined as the Re-Prefill Phase. Offloading this shared prefix cache to secondary storage is essential for memory scalability. Re-Prefill with offloading suffers from severe I/O bottlenecks in two aspects. First, semantic-aware KV cache pruning algorithms select important tokens in fine granularity, while systems manage I/O in coarse, fixed-size blocks, causing severe read amplification. Second, the sequential dependency between identifying important tokens and loading KV cache creates idle I/O and compute bubbles, under-utilizing system resources. This paper proposes textitContiguousKV, a high-performance prefix KV cache offloading system that bridges algorithmic semantics with I/O efficiency to accelerate the Re-Prefill phase. We first introduce textitContiguousChunk, a unified data management granularity that aligns KV cache pruning with I/O operations. All the mechanisms critical for I/O performance are performed at the granularity of ContiguousChunk, thereby eliminating read amplification. By exploiting the high similarity in important ContiguousChunk indices across layers, we propose intra- and inter-period asynchronous prefetching to break the sequential dependency between I/O and compute, effectively eliminating idle bubbles. Finally, we propose attention-guided cache management to retain semantically critical prefix data in memory. Evaluations on Qwen2.5 series models show that ContiguousKV achieves a 3.85x speedup in the Re-Prefill phase over the state-of-the-art offloading system IMPRESS, while maintaining high output quality.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 53 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

43 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
23 more
  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  2. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  4. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
  5. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  6. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  7. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  8. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  9. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  10. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  11. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  12. 2024SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generationcited by 40
  13. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  14. 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Modelscited by 34
  15. 2024When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environmentscited by 25
  16. 2024M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluationcited by 21
  17. 2024RecAI: Leveraging Large Language Models for Next-Generation Recommender Systemscited by 20
  18. 2024Inferring Functionality of Attention Heads from their Parameterscited by 17
  19. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 15
  20. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compressioncited by 7
  21. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
  22. 2024ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusioncited by 5
  23. 2023Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPTcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference
  2. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
  3. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  8. 2026KV Cache Offloading for Context-Intensive Tasks