mlsanityData to 7 Oct 2026

Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration

arXiv 2510.03865 ↗First posted 4 Oct 2025cs.LG cs.AI cs.CLcited by 7

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has recently enhanced the reasoning capabilities of large language models (LLMs), particularly for mathematical problem solving. However, a fundamental limitation remains: as the sampling budget increases, the advantage of RLVR-trained models over their pretrained bases often diminishes or even vanishes, revealing a strong dependence on the base model's restricted search space. We attribute this phenomenon to the widespread use of the reverse Kullback-Leibler (KL) divergence regularizer, whose mode-seeking behavior keeps the policy trapped inside the base model's support region and hampers wider exploration. To address this issue, we propose RAPO (Rewards-Aware Policy Optimization), an algorithm to promote broader yet focused exploration. Our method (i) utilizes the forward KL penalty to replace the reverse KL penalty for out-of-distribution exploration, and (ii) reweights the reference policy to facilitate adaptive in-distribution exploration. We train Qwen2.5-3B and 7B models with RAPO on the 8K SimpleRL-Zero dataset, without supervised fine-tuning, and evaluate them on AIME2024 and AIME2025. Results show that RAPO consistently improves problem-solving performance. Notably, RAPO enables models to surpass the base model's performance ceiling and solves previously intractable problems, advancing the frontier of RLVR for challenging reasoning tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 35 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

31 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025LIMR: Less is More for RL Scaling17 Feb
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  9. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  10. 2023Let's Verify Step by Step31 May
  11. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2025s1: Simple test-time scalingcited by 1,311
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
8 more
  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  4. 2024Token-level Direct Preference Optimizationcited by 135
  5. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  6. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  7. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  8. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 3
  2. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
  3. 2025Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Modelscited by 1
  4. 2025Towards an AI Fluid Scientist: LLM-Powered Scientific Discovery in Experimental Fluid Mechanicscited by 1
  5. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samplescited by 0
  6. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 0
  7. 2026Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  3. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  4. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
  5. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  7. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  8. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training