# ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning

arXiv: https://arxiv.org/abs/2602.09953 · HTML: https://mlsanity.com/paper/2602.09953/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning](https://mlsanity.com/paper/2501.12570/) (2025)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [CoT-Valve: Length-Compressible Chain-of-Thought Tuning](https://mlsanity.com/paper/2502.09601/) (2025)
- [AdaptThink: Reasoning Models Can Learn When to Think](https://mlsanity.com/paper/2505.13417/) (2025)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models](https://mlsanity.com/paper/2505.07686/) (2025)
- [Learn to Reason Efficiently with Adaptive Length-based Reward Shaping](https://mlsanity.com/paper/2505.15612/) (2025)
- [ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning](https://mlsanity.com/paper/2504.21370/) (2025)
- [Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL](https://mlsanity.com/paper/2505.10832/) (2025)
- [Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning](https://mlsanity.com/paper/2506.05256/) (2025)
- [Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning](https://mlsanity.com/paper/2510.10649/) (2025)
- [Promoting Efficient Reasoning with Verifiable Stepwise Reward](https://mlsanity.com/paper/2508.10293/) (2025)
- [ExpSeek: Self-Triggered Experience Seeking for Web Agents](https://mlsanity.com/paper/2601.08605/) (2026)
- [Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning](https://mlsanity.com/paper/2505.16315/) (2025)
- [The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis](https://mlsanity.com/paper/2508.17627/) (2025)
- [Think Clearly: Improving Reasoning via Redundant Token Pruning](https://mlsanity.com/paper/2507.08806/) (2025)
- [Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think](https://mlsanity.com/paper/2504.20708/) (2025)

## Built on by (most cited first, 9 of 9)

- [MemPO: Self-Memory Policy Optimization for Long-Horizon Agents](https://mlsanity.com/paper/2603.00680/) (2026)
- [FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Models](https://mlsanity.com/paper/2604.02967/) (2026)
- [KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance](https://mlsanity.com/paper/2604.12627/) (2026)
- [Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping](https://mlsanity.com/paper/2603.23998/) (2026)
- [Dissecting Failure Dynamics in Large Language Model Reasoning](https://mlsanity.com/paper/2604.14528/) (2026)
- [SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoning](https://mlsanity.com/paper/2604.06636/) (2026)
- [Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning](https://mlsanity.com/paper/2605.05226/) (2026)
- [Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents](https://mlsanity.com/paper/2605.08721/) (2026)
- [Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization](https://mlsanity.com/paper/2609.39402/) (2026)

## Abstract (arXiv)

Large reasoning models trained with reinforcement learning and verifiable rewards (RLVR) achieve strong performance on complex reasoning tasks, yet often overthink, generating redundant reasoning without performance gains. Existing trajectory-level length penalties often fail to effectively shorten reasoning length and degrade accuracy, as they uniformly treat all reasoning steps and lack fine-grained signals to distinguish redundancy from necessity. Meanwhile, process-supervised methods are typically resource-intensive and suffer from inaccurate credit assignment. To address these issues, we propose ATTNPO, a low-overhead process-supervised RL framework that leverages the model's intrinsic attention signals for step-level credit assignment. We first identify a set of special attention heads that naturally focus on essential steps while suppressing redundant ones. By leveraging the attention scores of these heads, We then employ two sub-strategies to mitigate overthinking by discouraging redundant steps while preserving accuracy by reducing penalties on essential steps. Experimental results show that ATTNPO substantially reduces reasoning length while significantly improving performance across 9 benchmarks.