# SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

arXiv: https://arxiv.org/abs/2506.08889 · HTML: https://mlsanity.com/paper/2506.08889/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 63 of its 78 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025)

## Built on by (most cited first, 30 of 30)

- [IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse](https://mlsanity.com/paper/2603.12201/) (2026)
- [Efficient Attention Mechanisms for Large Language Models: A Survey](https://mlsanity.com/paper/2507.19595/) (2025)
- [Bidirectional Sparse Attention for Faster Video Diffusion Training](https://mlsanity.com/paper/2509.01085/) (2025)
- [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://mlsanity.com/paper/2512.03324/) (2025)
- [BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding](https://mlsanity.com/paper/2512.12087/) (2025)
- [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://mlsanity.com/paper/2605.09649/) (2026)
- [Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference](https://mlsanity.com/paper/2512.16391/) (2025)
- [HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing](https://mlsanity.com/paper/2602.03560/) (2026)
- [LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards](https://mlsanity.com/paper/2602.05758/) (2026)
- [Value-Aware Stochastic KV Cache Eviction for Reasoning Models](https://mlsanity.com/paper/2606.03928/) (2026)
- [R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning](https://mlsanity.com/paper/2507.17307/) (2025)
- [DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning](https://mlsanity.com/paper/2510.09883/) (2025)
- [SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space](https://mlsanity.com/paper/2511.20102/) (2025)
- [Hold Onto That Thought: Assessing KV Cache Compression On Reasoning](https://mlsanity.com/paper/2512.12008/) (2025)
- [LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding](https://mlsanity.com/paper/2602.04541/) (2026)
- [DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention](https://mlsanity.com/paper/2605.18753/) (2026)
- [An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference](https://mlsanity.com/paper/2605.07719/) (2026)
- [PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention](https://mlsanity.com/paper/2607.24593/) (2026)
- [You Only Index Once: Cross-Layer Sparse Attention with Shared Routing](https://mlsanity.com/paper/2606.06467/) (2026)
- [SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking](https://mlsanity.com/paper/2609.13141/) (2026)
- [Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies](https://mlsanity.com/paper/2511.00819/) (2025)
- [Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning](https://mlsanity.com/paper/2609.03430/) (2026)
- [Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference](https://mlsanity.com/paper/2603.29002/) (2026)
- [BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference](https://mlsanity.com/paper/2609.04971/) (2026)
- [On-Demand Attention: Language Models Know When to Recall](https://mlsanity.com/paper/2609.20734/) (2026)
- [The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends](https://mlsanity.com/paper/2609.39661/) (2026)
- [HHR: Hierarchical Hash Retrieval for Efficient LLM Generation](https://mlsanity.com/paper/2610.01230/) (2026)
- [MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding](https://mlsanity.com/paper/2610.04313/) (2026)
- [Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning](https://mlsanity.com/paper/2610.05685/) (2026)
- [Joint Subspace Sparse Attention for Efficient Long-Context Decoding](https://mlsanity.com/paper/2610.19026/) (2026)

## Abstract (arXiv)

We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at: https://github.com/microsoft/SeerAttention.