mlsanityData to 14 Oct 2026

Hardware-Efficient Attention for Fast Decoding

arXiv 2505.21487 ↗First posted 27 May 2025cs.LG cs.CLcited by 10

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLM decoding is bottlenecked for large batches and long contexts by loading the key-value (KV) cache from high-bandwidth memory, which inflates per-token latency, while the sequential nature of decoding limits parallelism. We analyze the interplay among arithmetic intensity, parallelization, and model quality and question whether current architectures fully exploit modern hardware. This work redesigns attention to perform more computation per byte loaded from memory to maximize hardware efficiency without trading off parallel scalability. We first propose Grouped-Tied Attention (GTA), a simple variant that combines and reuses key and value states, reducing memory transfers without compromising model quality. We then introduce Grouped Latent Attention (GLA), a parallel-friendly latent attention paired with low-level optimizations for fast decoding while maintaining high model quality. Experiments show that GTA matches Grouped-Query Attention (GQA) quality while using roughly half the KV cache and that GLA matches Multi-head Latent Attention (MLA) and is easier to shard. Our optimized GLA kernel is up to 2× faster than FlashMLA, for example, in a speculative decoding setting when the query length exceeds one. Furthermore, by fetching a smaller KV cache per device, GLA reduces end-to-end latency and increases throughput in online serving benchmarks by up to 2×.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 45 of its 73 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

45 of 73 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs20 Feb
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  4. 2024Effectively Compress KV Heads for LLM11 Jun
  5. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
27 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  5. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2022Efficiently Scaling Transformer Inferencecited by 539
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  13. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  15. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  16. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  17. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  18. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  19. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  20. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 59
  21. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  22. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  23. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  24. 2025Tensor Product Attention Is All You Needcited by 31
  25. 2024Weight decay induces low-rank attention layerscited by 27
  26. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  27. 2023Reducing shared memory footprint to leverage high throughput on Tensor Cores and its flexible API extension librarycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  2. 2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
  3. 2026Multi-Head Low-Rank Attentioncited by 2
  4. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  5. 2025AMLA: MUL by ADD in FlashAttention Rescalingcited by 1
  6. 2025Knocking-Heads Attentioncited by 1
  7. 2026GQLA: Group-Query Latent Attention for Hardware-Adaptive Decodingcited by 1
  8. 2025Exact Causal Attention with 10% Fewer Operationscited by 0
  9. 2026SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Modelscited by 0
  10. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel
  2. 2026SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding
  3. 2025GTA: Grouped-head latenT Attention
  4. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
  5. 2025CoDec: Prefix-Shared Decoding Kernel for LLMs
  6. 2026Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
  7. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
  8. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding