mlsanityData to 7 Oct 2026

Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference

arXiv 2502.15294 ↗First posted 21 Feb 2025cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The increasing context window size in large language models (LLMs) has improved their ability to handle complex, long-text tasks. However, as the conversation rounds continue, it is required to store a large amount of KV cache in GPU memory, which significantly affects the efficiency and even availability of the model serving systems. This paper analyzes dialogue data from real users on the granularity of round and discovers that the LLM inference manifests a watershed layer, after which the distribution of round-level attention shows notable similarity. Based on this, we propose Round Attention - a novel round-level attention mechanism that selectively processes the KV cache of top-k relevant rounds, where k is dynamically determined through the attention matrix in the watershed layer. Theoretical analysis demonstrates that our method reduces memory usage by 54% to 82%, while experimental results confirm that loading sparse critical-round KV cache maintains answer accuracy without performance degradation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 25 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

22 of 25 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity3 Dec
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024Cross-layer Attention Sharing for Pre-trained Large Language Models4 Aug
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 14

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  6. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  7. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  8. 2024Transformers are Multi-State RNNscited by 120
  9. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  10. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  11. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  12. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
2 more
  1. 2019Sharing Attention Weights for Fast Transformercited by 34
  2. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  4. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  5. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  6. 2026CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
  7. 2025LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues
  8. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving