mlsanityData to 14 Oct 2026

Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration

arXiv 2511.00794 ↗First posted 2 Nov 2025cs.LG cs.AIcited by 7

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models, yet training remains costly because many rollouts contribute little to optimization, considering the amount of computation required. This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR. We propose PREPO with two complementary components. First, we adopt prompt perplexity as an indicator of model adaptability in learning, enabling the model to progress from well-understood contexts to more challenging ones. Second, we amplify the discrepancy among the rollouts by differentiating their relative entropy, and prioritize sequences that exhibit a higher degree of exploration. Together, these mechanisms reduce rollout demand while preserving competitive performance. On the Qwen and Llama models, PREPO achieves effective results on mathematical reasoning benchmarks with up to 3 times fewer rollouts than the baselines. Beyond empirical gains, we provide theoretical and in-depth analyses explaining the underlying rationale of our method to improve the data efficiency of RLVR.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 31 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

26 of 31 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  2. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization9 Jul
  3. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  4. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
  5. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  6. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  10. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation13 May
  11. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  12. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
4 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2023Let's Verify Step by Step31 May

Other references · 10

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Difficulty-Estimated Policy Optimizationcited by 3
  2. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  3. 2026DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Offcited by 0
  4. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  5. 2026Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoningcited by 0
  6. 2026Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimizationcited by 0
  7. 2026CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  2. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example
  4. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  5. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
  6. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  8. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models