mlsanityData to 14 Oct 2026

Reinforcing General Reasoning without Verifiers

arXiv 2505.21493 ↗First posted 27 May 2025cs.LG cs.CLcited by 77

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method (VeriFree) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 48 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

34 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  2. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains6 May
  3. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  10. 2023Let's Verify Step by Step31 May
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
10 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2024Self-Rewarding Language Modelscited by 681
  3. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  4. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  5. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  6. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  7. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  8. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 55
  9. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  10. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 40; 2026: 37.

77 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 100
  4. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 85
  5. 2025Self-Questioning Language Modelscited by 50
  6. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  7. 2025Reinforcement Pre-Trainingcited by 40
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  9. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  10. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  11. 2025RLP: Reinforcement as a Pretraining Objectivecited by 25
  12. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Densecited by 20
38 more
  1. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  2. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  3. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 18
  4. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  5. 2026G-Zero: Self-Play for Open-Ended Generation from Zero Datacited by 17
  6. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
  7. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  8. 2026VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reductioncited by 10
  9. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  10. 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewardingcited by 9
  11. 2026Learning to Simulate Human Dialoguecited by 9
  12. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  13. 2025Variational Reasoning for Language Modelscited by 8
  14. 2025RAVR: Reference-Answer-guided Variational Reasoning for Large Language Modelscited by 8
  15. 2025Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesiscited by 8
  16. 2025PretrainZero: Reinforcement Active Pretrainingcited by 8
  17. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  18. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  19. 2025Escaping the Verifier: Learning to Reason via Demonstrationscited by 7
  20. 2026Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clusteringcited by 7
  21. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 7
  22. 2026From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generationcited by 6
  23. 2025URPO: A Unified Reward & Policy Optimization Framework for Large Language Modelscited by 5
  24. 2025Correct Reasoning Paths Visit Shared Decision Pivotscited by 5
  25. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  26. 2025Your thoughts tell who you are: Characterize the reasoning patterns of LRMscited by 4
  27. 2025Rethinking Reasoning with MDLMs: Early Exits, Post-hoc Reasoning, and Beyondcited by 4
  28. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
  29. 2025Rectifying LLM Thought from Lens of Optimizationcited by 4
  30. 2026Improving Data and Reward Design for Scientific Reasoning in Large Language Modelscited by 4
  31. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 4
  32. 2026Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillationcited by 4
  33. 2025Learning to Reason as Action Abstractions with Scalable Mid-Training RLcited by 3
  34. 2025CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Samplingcited by 3
  35. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 3
  36. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 3
  37. 2025Coupled Variational Reinforcement Learning for Language Model General Reasoningcited by 2
  38. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
  2. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
  3. 2025CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  5. 2026GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization
  6. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example
  8. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving