# RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning

arXiv: https://arxiv.org/abs/2502.11147 · HTML: https://mlsanity.com/paper/2502.11147/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 0 of its 52 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Built on by (most cited first, 18 of 18)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models](https://mlsanity.com/paper/2510.01290/) (2025)
- [TriAttention: Efficient Long Reasoning with Trigonometric KV Compression](https://mlsanity.com/paper/2604.04921/) (2026)
- [Kinetics: Rethinking Test-Time Scaling Laws](https://mlsanity.com/paper/2506.05333/) (2025)
- [SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning](https://mlsanity.com/paper/2602.22603/) (2026)
- [DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning](https://mlsanity.com/paper/2510.09883/) (2025)
- [You Need an Encoder for Native Position-Independent Caching](https://mlsanity.com/paper/2602.01519/) (2026)
- [Sparse Attention across Multiple-context KV Cache](https://mlsanity.com/paper/2508.11661/) (2025)
- [SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models](https://mlsanity.com/paper/2512.07993/) (2025)
- [LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequences](https://mlsanity.com/paper/2510.11292/) (2025)
- [ReasonCache: Accelerating Large Reasoning Model Serving through KV Cache Sharing](https://mlsanity.com/paper/2507.21433/) (2025)
- [CSAttention: Centroid-Scoring Attention for Accelerating LLM Inference](https://mlsanity.com/paper/2604.08584/) (2026)
- [Epiphany-Aware KV Cache Eviction Without the Attention Matrix](https://mlsanity.com/paper/2606.26472/) (2026)
- [Prefix Sliding for efficient test-time scaling](https://mlsanity.com/paper/2608.26070/) (2026)
- [GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving](https://mlsanity.com/paper/2609.03494/) (2026)
- [StepKV: Step-Aware KV Cache Compression for LLM Agents](https://mlsanity.com/paper/2609.22158/) (2026)
- [ActKV: Efficient LLM Agents through Action-Guided KV Cache Management](https://mlsanity.com/paper/2609.31395/) (2026)
- [Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning](https://mlsanity.com/paper/2610.05685/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated strong capabilities across various domains, with recent advancements in challenging reasoning tasks such as mathematics and programming. However, solving reasoning tasks often requires an LLM to generate long sequences, incurring O(N) time and memory complexities per token, where N is the current sequence length. To reduce complexities, existing sparsity-based algorithms propose to retain Key-Value (KV) vectors, the intermediate representations of only the most critical tokens. However, these algorithms struggle with the "impossible trinity" of accuracy, time, and memory. For example, the state-of-the-art algorithm, Quest, achieves high accuracy with O(L) time but O(N) memory (L is the cache budget, L ll N). To address the "impossible trinity", in this paper, we identify a new attention pattern during the decode stage of reasoning tasks, where milestone tokens (analogous to lemmas in mathematical proofs) emerge, are utilized, and then become unimportant afterward. Based on this pattern, we propose a new algorithm RaaS that identifies milestone tokens and retains their KV vectors until they are no longer needed, achieving high accuracy with O(L) time and O(L) memory complexities.