KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
25 of 35 references.
References · 25
- 2017Attention Is All You Needcited by 59,932
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
13 more
- 2024Transformers are Multi-State RNNscited by 120
- 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
- 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
- 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
- 2024Block-Attention for Efficient Prefillingcited by 34
- 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
- 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
- 2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 14
- 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 14
- 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generationcited by 9
- 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 7
- 2025Value-Guided KV Compression for LLMs via Approximated CUR Decompositioncited by 6
- 2025Sparse Attention across Multiple-context KV Cachecited by 5
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
- 2026Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Servingcited by 3
- 2026SemPIC: Learning Semantic Position-Independent KV Cachescited by 2
- 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
- 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 0
- 2026Where Should a Document Live: Context, Representations, or Parameters?cited by 0
- 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reusecited by 0
- 2026CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusioncited by 0
- 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptationcited by 0
- 2026Cache the Encoder Within:Compact, Reusable Memory across LLM Queriescited by 0
- 2026Cache the Encoder Within:Compact, Reusable Memory across LLM Queries7 Oct 2026
- 2026ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation29 Sep 2026
- 2026CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion28 Sep 2026
- 2026RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse27 Sep 2026
- 2026Where Should a Document Live: Context, Representations, or Parameters?15 Sep 2026
- 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints9 Sep 2026
- 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals30 Jul 2026
- 2026SemPIC: Learning Semantic Position-Independent KV Caches30 Jul 2026
- 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching1 Jul 2026
- 2026Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving7 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
- 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
- 2026Recency/Frequency Adaptive KV Caching for Large Language Model Serving
- 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
- 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
- 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2026KV Cache Offloading for Context-Intensive Tasks