mlsanityData to 14 Oct 2026

RLPR: Extrapolating RLVR to General Domains without Verifiers

arXiv 2506.18254 ↗First posted 23 Jun 2025cs.LG cs.AI cs.CLcited by 85

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising potential in advancing the reasoning capabilities of LLMs. However, its success remains largely confined to mathematical and code domains. This primary limitation stems from the heavy reliance on domain-specific verifiers, which results in prohibitive complexity and limited scalability. To address the challenge, our key observation is that LLM's intrinsic probability of generating a correct free-form answer directly indicates its own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer). Building on this insight, we propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains. RLPR uses the LLM's own token probability scores for reference answers as the reward signal and maximizes the expected reward during training. We find that addressing the high variance of this noisy probability reward is crucial to make it work, and propose prob-to-reward and stabilizing methods to ensure a precise and stable reward from LLM intrinsic probabilities. Comprehensive experiments in four general-domain benchmarks and three mathematical benchmarks show that RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models. Notably, RLPR outperforms concurrent VeriFree by 7.6 points on TheoremQA and 7.5 points on Minerva, and even surpasses strong verifier-model-dependent approaches General-Reasoner by 1.6 average points across seven benchmarks.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 31 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

20 of 31 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  2. 2025Reinforcing General Reasoning without Verifiers27 May
  3. 2025Learning to Reason without External Rewards26 May
  4. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  5. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  6. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains6 May
  7. 2025RM-R1: Reward Modeling as Reasoning5 May
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Process Reinforcement through Implicit Rewards3 Feb

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2024Generative Reward Modelscited by 148
  8. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 55.

85 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  3. 2025Self-Questioning Language Modelscited by 50
  4. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  5. 2025RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agentscited by 25
  6. 2025Learning to Orchestrate Agents in Natural Language with the Conductorcited by 23
  7. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  8. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 19
  9. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  10. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
  11. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  12. 2026G-Zero: Self-Play for Open-Ended Generation from Zero Datacited by 17
38 more
  1. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  2. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 13
  3. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
  4. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 11
  5. 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewardingcited by 9
  6. 2026Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMscited by 9
  7. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  8. 2025Variational Reasoning for Language Modelscited by 8
  9. 2025RAVR: Reference-Answer-guided Variational Reasoning for Large Language Modelscited by 8
  10. 2026Learning Robust Reasoning through Guided Adversarial Self-Playcited by 8
  11. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
  12. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  13. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 7
  14. 2025StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Modelscited by 6
  15. 2026From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generationcited by 6
  16. 2025SERL: Self-Examining Reinforcement Learning on Open-Domaincited by 5
  17. 2025Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewardscited by 5
  18. 2025TRINITY: An Evolved LLM Coordinatorcited by 5
  19. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verificationcited by 5
  20. 2026StaRPO: Stability-Augmented Reinforcement Policy Optimizationcited by 5
  21. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoningcited by 5
  22. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  23. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Modelcited by 4
  24. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
  25. 2025ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learningcited by 4
  26. 2025Rectifying LLM Thought from Lens of Optimizationcited by 4
  27. 2025Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasetscited by 4
  28. 2026Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalizationcited by 4
  29. 2026Stable Adaptive Thinking via Advantage Shaping and Length-Aware Gradient Regulationcited by 4
  30. 2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
  31. 2025Pessimistic Verification for Open Ended Math Questionscited by 3
  32. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 3
  33. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 3
  34. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  35. 2026R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoningcited by 3
  36. 2025Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMscited by 2
  37. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoningcited by 2
  38. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  4. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  5. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  6. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  7. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  8. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models