mlsanityData to 7 Oct 2026

Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity

arXiv 2605.28640 ↗First posted 27 May 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficient inference is critical for long-context language models, where attention computation and KV-cache access dominate the cost. Recent work RAT+, introduces a recurrence-augmented attention backbone that enables flexible dilated attention at inference time. In this paper, we investigate whether this exponentially decaying memory can also improve existing query-aware sparse inference methods. Using representative methods including Quest, MoBA, and SnapKV, we show that RAT+ consistently improves accuracy over standard attention across sparse budgets on eight needle-in-a-haystack tasks. We validate these gains both on the released checkpoints from the RAT+ paper and on OLMo2-7B, which we continue pretraining with the added memory module for 10B tokens. Finally, we propose two hypotheses explaining why this memory module benefits query-aware sparse inference and design targeted experiments to support them.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 14 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

12 of 14 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference20 Feb
  2. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr

Other references · 8

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  6. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  8. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
  2. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  3. 2026Simplified Sparse Attention via Gist Tokens
  4. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
  5. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  6. 2026Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
  7. 2025Rectified Sparse Attention
  8. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning