mlsanityData to 14 Oct 2026

Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models

arXiv 2509.26628 ↗First posted 30 Sep 2025cs.LG cs.CLcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 67 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

41 of 67 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  3. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment4 Aug
  4. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner31 Jul
  5. 2025Agentic Reinforced Policy Optimization26 Jul
  6. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  7. 2025First Return, Entropy-Eliciting Explore9 Jul
  8. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning2 Jul
  9. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs23 Jun
  10. 2025TreeRPO: Tree Relative Policy Optimization5 Jun
  11. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
14 more
  1. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  2. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  3. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning21 May
  4. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning21 Apr
  5. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Process Reinforcement through Implicit Rewards3 Feb
  13. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  14. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2025Group Sequence Policy Optimizationcited by 835
  8. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  9. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  10. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  11. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 129
  12. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
3 more
  1. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 75
  2. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Computecited by 38
  3. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  2. 2026GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughtscited by 17
  3. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  4. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  5. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 5
  6. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  7. 2026SD-E2: Semantic Exploration for Reasoning Under Token Budgetscited by 3
  8. 2025Robust Preference Alignment via Directional Neighborhood Consensuscited by 2
  9. 2026Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVRcited by 2
  10. 2026Smart Picks in the Dark: Towards Efficient RLVR for Reasoning via Tracing Metacognitive Pivotscited by 1
  11. 2026When Does In-Context Search Help? A Sampling-Complexity Theory of Reflection-Driven Reasoningcited by 1
  12. 2026Deep Dense Exploration for LLM Reinforcement Learning via Pivot-Driven Resamplingcited by 0
1 more
  1. 2026Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
  2. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  3. 2026ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning
  4. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  5. 2025Training Language Models to Reason Efficiently
  6. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  7. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning
  8. 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning