mlsanityData to 14 Oct 2026

SparQ Attention: Bandwidth-Efficient LLM Inference

arXiv 2312.04985 ↗First posted 8 Dec 2023cs.LGcited by 119

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The computational difficulties of large language model (LLM) inference remain a significant obstacle to their widespread deployment. The need for many applications to support long input sequences and process them in large batches typically causes token-generation to be bottlenecked by data transfer. For this reason, we introduce SparQ Attention, a technique for increasing the inference throughput of LLMs by utilising memory bandwidth more efficiently within the attention layers, through selective fetching of the cached history. Our proposed technique can be applied directly to off-the-shelf LLMs during inference, without requiring any modification to the pre-training setup or additional fine-tuning. We show that SparQ Attention brings up to 8x savings in attention data transfers without substantial drops in accuracy, by evaluating Llama 2 and 3, Mistral, Gemma and Pythia models on a wide range of downstream tasks.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 66 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

35 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2023Mistral 7Bcited by 3,164
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
21 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2020Reformer: The Efficient Transformercited by 1,668
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  10. 2020Efficient Transformers: A Surveycited by 790
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  13. 2022Efficiently Scaling Transformer Inferencecited by 539
  14. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  15. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  16. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  17. 2020Sparse Sinkhorn Attentioncited by 251
  18. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  19. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  20. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  21. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 39; 2026: 55.

119 papers.

Sort:

By area:

  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  3. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  6. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  8. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  9. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  10. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  11. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  12. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
38 more
  1. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  2. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  3. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 51
  4. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 51
  5. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  6. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  7. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 44
  8. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  9. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  10. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  11. 2025Tensor Product Attention Is All You Needcited by 31
  12. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  13. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  14. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  15. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
  16. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
  17. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  18. 2026SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuningcited by 18
  19. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 16
  20. 2026Learning to Evict from Key-Value Cachecited by 14
  21. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  22. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  23. 2024A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attentioncited by 12
  24. 2024RefreshKV: Updating Small KV Cache During Long-form Generationcited by 11
  25. 2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 11
  26. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimizationcited by 10
  27. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 10
  28. 2024SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Modelscited by 9
  29. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 9
  30. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  31. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  32. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  33. 2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inferencecited by 7
  34. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 7
  35. 2024LoMA: Lossless Compressed Memory Attentioncited by 6
  36. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  37. 2025Efficient Pretraining Length Scalingcited by 6
  38. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 6

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  3. 2024Star Attention: Efficient LLM Inference over Long Sequences
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insights
  5. 2024A Survey on Efficient Inference for Large Language Models
  6. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  7. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  8. 2024Beyond KV Caching: Shared Attention for Efficient LLMs