Data to 14 Oct 2026

KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse

arXiv 2502.16002 ↗First posted 21 Feb 2025cs.CLcited by 66

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

We describe KVLink, an approach for efficient key-value (KV) cache reuse in large language models (LLMs). In many LLM applications, different inputs can share overlapping context, such as the same retrieved document appearing in multiple queries. However, the LLMs still need to encode the entire context for each query, leading to redundant computation. In this paper, we investigate a new strategy to eliminate such inefficiency, where the KV cache of each document is precomputed independently. During inference, the KV caches of retrieved documents are concatenated, allowing the model to reuse cached representations instead of recomputing them. To mitigate the performance degradation when using KV caches computed independently for each document, KVLink introduces two key techniques: adjusting positional embeddings of the KV cache at inference to match the global position after concatenation, and using trainable special tokens to restore self-attention across independently encoded documents. Experiments across 7 datasets demonstrate that KVLink improves question answering accuracy by an average of 4% over state-of-the-art methods. Furthermore, by leveraging precomputed KV caches, our approach reduces time-to-first-token by up to 96% compared to standard LLM inference, making it a scalable and efficient solution for context reuse. Additionally, KVLink can be combined with KV cache compression to further save cache loading and storage overhead while outperforming the baselines.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 61 of its 84 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

61 of 84 references.

Closest ancestors · 4

Same direction, newest first

Other references · 57

2023GPT-4 Technical Reportcited by 18,396
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2020Measuring Massive Multitask Language Understandingcited by 7,963
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
2023Efficient Streaming Language Models with Attention Sinkscited by 2,186

38 more

2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
2017Crowdsourcing Multiple Choice Science Questionscited by 765
2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
2023Augmenting Language Models with Long-Term Memorycited by 189
2023Speculative Decoding with Big Little Decodercited by 164
2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
2024Transformers are Multi-State RNNscited by 120
2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
2020Learning Dense Representations of Phrases at Scalecited by 91

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 54.

66 papers.

Sort:

By area:

2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 27
2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 16
2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reusecited by 8

38 more

2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
2026You Need an Encoder for Native Position-Independent Cachingcited by 6
2026Simplified Sparse Attention via Gist Tokenscited by 6
2026KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasingcited by 6
2026Models Take Notes at Prefill: KV Cache Can Be Editable and Composablecited by 6
2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 5
2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
2026An experimental study of KV cache reuse strategies in chunk-level caching systemscited by 5
2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inferencecited by 4
2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
2026TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agentscited by 3
2026InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Contextcited by 3
2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
2026LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMscited by 3
2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agentscited by 3
2026When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judgescited by 2
2026TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQLcited by 2
2026HijackKV: New Threat in Position-Independent KV Cache Reusecited by 2
2026Compute Globally, Materialize Locally: The Memory Contract of Sparse Event-KVcited by 2
2026SemPIC: Learning Semantic Position-Independent KV Cachescited by 2
2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1
2026Parallel Context-of-Experts Decoding for Retrieval Augmented Generationcited by 1
2026GORGO: Online Tuning for Cross-Region Network-Aware LLM Servingcited by 1
2026SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariancecited by 1
2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 1
2025CIFLEX: Contextual Instruction Flow for Sub-task Execution in Multi-Turn Interactions with a Single On-Device LLMcited by 0
2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
2026HUOZIIME: An On-Device LLM-enhanced Input Method for Deep Personalizationcited by 0
2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attentioncited by 0
2026Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflowscited by 0
2026ChronoStitch: Training-Free Composition of Visual KV Memories for Long-Horizon Temporal Reasoningcited by 0
2026CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAGcited by 0
2026MaSRead: Content-Addressed Reading of Replicated Latent Storescited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way