mlsanityData to 14 Oct 2026

LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

arXiv 2606.04302 ↗First posted 3 Jun 2026cs.CL cs.LGcited by 2

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Key-value (KV) caching accelerates inference of large language models (LLMs) by reusing past computations for generated tokens. Its importance becomes even greater in long-context applications such as retrieval-augmented generation (RAG) and in-context learning (ICL). However, conventional KV caching embeds positional information directly into the cache, limiting its reusability. Existing solutions either restrict reuse to prefixes or require expensive memory materialization for positional re-encoding. We introduce LazyAttention, a novel attention mechanism that kernelizes deferred positional encoding to enable zero-copy, position-agnostic KV reuse. By adjusting positional encoding within attention kernels on-the-fly, LazyAttention resolves the materialization bottleneck, allowing a single physical KV copy to serve multiple logical requests at arbitrary positions. Leveraging attention kernels tailored for prefilling and decoding, our system achieves significant efficiency improvements: under skewed document distributions, it reduces time-to-first-token (TTFT) by 1.37× and increases inference throughput by 1.40× compared to the state-of-the-art Block-Attention, while maintaining comparable output quality.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 43 of its 57 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

43 of 57 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Serving18 Dec
  2. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks20 Dec
  3. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
  4. 2024Block-Attention for Efficient Prefilling14 Sep
  5. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation18 Apr
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context2 Oct
  7. 2023In-Context Retrieval-Augmented Language Models31 Jan
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  10. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
22 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  3. 2023The Falcon Series of Open Language Modelscited by 535
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  7. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  8. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  9. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  10. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  11. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  12. 2022Large Language Models with Controllable Working Memorycited by 169
  13. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  14. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  15. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  16. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  17. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
  18. 2024Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attentioncited by 31
  19. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  20. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applicationscited by 22
  21. 2025PBE Meets LLM: When Few Examples Aren't Few-Shot Enoughcited by 3
  22. 2025Large-scale Evaluation of Notebook Checkpointing with AI Agentscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Efficient Iterative Retrieval with Heterogeneous Batchingcited by 1
  2. 2026PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation
  2. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  3. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  4. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  5. 2026Where Matters More Than What: Decoding-aligned KV Cache Compression via Position-aware Pseudo Queries
  6. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
  7. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  8. 2026ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction