mlsanityData to 14 Oct 2026

RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization

arXiv 2508.00222 ↗First posted 31 Jul 2025cs.AI cs.CL cs.LGcited by 35

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). However, it struggles to break through the inherent capability boundaries of the base LLM, due to its essentially on-policy strategy coupled with LLM's immense action space and sparse reward. Critically, RLVR can lead to the capability boundary collapse, narrowing the LLM's problem-solving scope. To address this problem, we propose RL-PLUS, a novel hybrid-policy optimization approach for LLMs that synergizes internal exploitation with external data to achieve stronger reasoning capabilities and surpass the boundaries of base models. RL-PLUS integrates two core components, i.e., Multiple Importance Sampling to address distributional mismatch from external data, and Exploration-Based Advantage Function to guide the model towards high-value, unexplored reasoning paths. We provide both theoretical analysis and extensive experiments to demonstrate the superiority and generalizability of our approach. Compared with existing RLVR methods, RL-PLUS achieves 1) state-of-the-art performance on six math reasoning benchmarks; 2) superior performance on six out-of-distribution reasoning tasks; 3) consistent and significant gains across diverse model families, with average relative improvements up to 69.2%. Moreover, the analysis of Pass@k curves indicates that RL-PLUS effectively resolves the capability boundary collapse problem.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 44 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

35 of 44 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  2. 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning1 Jun
  3. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning30 May
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  5. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning21 May
  6. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs19 May
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  8. 2025Learning to Reason under Off-Policy Guidance21 Apr
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  11. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
7 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
4 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  3. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  4. 2024UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Functioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 19.

35 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  3. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  4. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  5. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  6. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  7. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 24
  8. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  9. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learningcited by 17
  10. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  11. 2025Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Modelcited by 13
  12. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
23 more
  1. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  2. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  3. 2025Emergent Slow Thinking in LLMs as Inverse Tree Freezingcited by 8
  4. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 6
  5. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewardscited by 6
  6. 2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
  7. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  8. 2026The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVRcited by 3
  9. 2026ICPO: Illocution-Calibrated Policy Optimization for Multi-Turn Conversationcited by 2
  10. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  11. 2025MCCE: A Framework for Multi-LLM Collaborative Search in Discrete Spaces with Similarity-Filtered Preference Learningcited by 1
  12. 2026Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoningcited by 1
  13. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVRcited by 1
  14. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  15. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM trainingcited by 1
  16. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branchescited by 1
  17. 2026Capsule Lens: Locating and Tracking Concept Geometry in Model Representationscited by 1
  18. 2025Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategiescited by 0
  19. 2026To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregationcited by 0
  20. 2026Preference Tuning as Spectral Update Reorganizationcited by 0
  21. 2026Reinforcement Learning for Symbolic Equation Solvingcited by 0
  22. 2026Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVRcited by 0
  23. 2026SPROUT: Scaling Agentic Reinforcement Learning for Long-Horizon Software Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  3. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  5. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  6. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  8. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation