mlsanityData to 7 Oct 2026

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention

arXiv 2406.15486 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) now support extremely long context windows, but the quadratic complexity of vanilla attention results in significantly long Time-to-First-Token (TTFT) latency. Existing approaches to address this complexity require additional pretraining or finetuning, and often sacrifice model accuracy. In this paper, we first provide both theoretical and empirical foundations for near-lossless sparse attention. We find dynamically capturing head-specific sparse patterns at runtime with low overhead is crucial. To address this, we propose SampleAttention, an adaptive structured and near-lossless sparse attention. Leveraging observed significant sparse patterns, SampleAttention attends to a fixed percentage of adjacent tokens to capture local window patterns, and employs a two-stage query-guided key-value filtering approach, which adaptively select a minimum set of key-values with low overhead, to capture column stripe patterns. Comprehensive evaluations show that SampleAttention can seamlessly replace vanilla attention in off-the-shelf LLMs with nearly no accuracy loss, and reduces TTFT by up to 2.42× compared with FlashAttention.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 59 of its 80 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

59 of 80 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  13. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  14. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  15. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  16. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  17. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  18. 2020Efficient Transformers: A Surveycited by 784
  19. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  20. 2024Yi: Open Foundation Models by 01.AIcited by 706
  21. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  22. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  23. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  24. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  25. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  26. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  27. 2024InternLM2 Technical Reportcited by 391
  28. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  29. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  30. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  31. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  32. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  33. 2022Recurrent Memory Transformercited by 197
  34. 2019MLPerf Inference Benchmarkcited by 195
  35. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  36. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  37. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  38. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 17.

39 papers.

Sort:

By area:

  1. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  2. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  3. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  4. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  5. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  6. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  7. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43
  8. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  9. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  10. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  11. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  12. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 12
27 more
  1. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  2. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  3. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  4. 2025FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernelcited by 7
  5. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  6. 2025vAttention: Verified Sparse Attentioncited by 6
  7. 2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generationcited by 5
  8. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  9. 2025PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generationcited by 4
  10. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  11. 2025AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Modelscited by 3
  12. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modelingcited by 3
  13. 2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3
  14. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  15. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  16. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  17. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  18. 2026CSAttention: Centroid-Scoring Attention for Accelerating LLM Inferencecited by 2
  19. 2026Fat-Cat: Document-Driven Metacognitive Multi-Agent System for Complex Reasoningcited by 1
  20. 2026RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inferencecited by 1
  21. 2026VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefillingcited by 1
  22. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systemscited by 1
  23. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cachecited by 1
  24. 2026SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariancecited by 1
  25. 2025Mitigating KV Cache Competition to Enhance User Experience in LLM Inferencecited by 0
  26. 2026ASAP: A Disaggregated and Asynchronous Inference System for MoE Prefillcited by 0
  27. 2026SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUscited by 0