mlsanityData to 14 Oct 2026

SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

arXiv 2506.08889 ↗First posted 10 Jun 2025cs.LG cs.AIcited by 30

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at: https://github.com/microsoft/SeerAttention.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 63 of its 78 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

63 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Models30 May
  2. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models7 Apr

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  13. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  15. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  16. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  17. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  18. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  19. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  20. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  21. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  22. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  23. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  24. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  25. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  26. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  27. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  28. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  29. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  30. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  31. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  32. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  33. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  34. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  35. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  36. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  37. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  38. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 20.

30 papers.

Sort:

By area:

  1. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 31
  2. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
  3. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 19
  4. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 19
  5. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 13
  6. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 11
  7. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 10
  8. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 10
  9. 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewardscited by 10
  10. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  11. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  12. 2025DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoningcited by 6
18 more
  1. 2025SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Spacecited by 6
  2. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoningcited by 6
  3. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decodingcited by 5
  4. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 4
  5. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  6. 2026PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attentioncited by 3
  7. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routingcited by 2
  8. 2026SAS: Simple Attention Sparsification via End-to-End Optimization of Context Rankingcited by 2
  9. 2025Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategiescited by 1
  10. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  11. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inferencecited by 0
  12. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inferencecited by 0
  13. 2026On-Demand Attention: Language Models Know When to Recallcited by 0
  14. 2026The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendscited by 0
  15. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0
  16. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0
  17. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0
  18. 2026Joint Subspace Sparse Attention for Efficient Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention
  2. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference
  3. 2026LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
  4. 2026Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding
  5. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
  6. 2025Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attention
  7. 2026Simplified Sparse Attention via Gist Tokens
  8. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modeling