mlsanityData to 7 Oct 2026

NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning

arXiv 2505.16022 ↗First posted 21 May 2025cs.CL cs.AI cs.LGcited by 15

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances such as DeepSeek R1-Zero highlight the effectiveness of incentive training, a reinforcement learning paradigm that computes rewards solely based on the final answer part of a language model's output, thereby encouraging the generation of intermediate reasoning steps. However, these methods fundamentally rely on external verifiers, which limits their applicability to domains like mathematics and coding where such verifiers are readily available. Although reward models can serve as verifiers, they require high-quality annotated data and are costly to train. In this work, we propose NOVER, NO-VERifier Reinforcement Learning, a general reinforcement learning framework that requires only standard supervised fine-tuning data with no need for an external verifier. NOVER enables incentive training across a wide range of text-to-text tasks and outperforms the model of the same size distilled from large reasoning models such as DeepSeek R1 671B by 7.7 percent. Moreover, the flexibility of NOVER enables new possibilities for optimizing large language models, such as inverse incentive training.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 63 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

32 of 63 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  8. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  9. 2023Let's Verify Step by Step31 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  11. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  12. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
10 more
  1. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  3. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  5. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  6. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  7. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  8. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  9. 2025Learning to Reason for Long-Form Story Generationcited by 34
  10. 2023Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?cited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Language Models that Think, Chat Bettercited by 23
  2. 2025RLP: Reinforcement as a Pretraining Objectivecited by 23
  3. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  4. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  5. 2026VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reductioncited by 9
  6. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  7. 2026Likelihood-Based Reward Designs for General LLM Reasoningcited by 6
  8. 2026Efficient Process Reward Modeling via Contrastive Mutual Informationcited by 4
  9. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2
  10. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  11. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifierscited by 1
  12. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedbackcited by 1
3 more
  1. 2026Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Driftcited by 1
  2. 2026Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilitiescited by 0
  3. 2026Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneckcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward
  2. 2025Reinforcing General Reasoning without Verifiers
  3. 2025Generalist Reward Models: Found Inside Large Language Models
  4. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  5. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  6. 2025Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense
  7. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  8. 2026GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings