Data to 7 Oct 2026

Generative Verifiers: Reward Modeling as Next-Token Prediction

arXiv 2408.15240 ↗First posted 27 Aug 2024cs.LGcited by 465

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs). A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected. While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs. To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. Compared to standard verifiers, such generative verifiers (GenRM) can benefit from several advantages of LLMs: they integrate seamlessly with instruction tuning, enable chain-of-thought reasoning, and can utilize additional test-time compute via majority voting for better verification. We demonstrate that GenRM outperforms discriminative, DPO verifiers, and LLM-as-a-Judge, resulting in large performance gains with Best-of-N, namely 5% rightarrow 45.3% on algorithmic tasks and 73% rightarrow 93.4% on GSM8K. In easy-to-hard generalization settings, we observe improvements of 28% rightarrow 44.6% on MATH, and 37.9% rightarrow 53.5% on MMLU abstract algebra. Furthermore, we find that training GenRM with synthetic verification rationales is sufficient to pick out subtle errors on math problems. Finally, we demonstrate that GenRM scales favorably with model size and test-time compute.

Key facts. Cited by 465 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 63 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

55 of 63 references.

Closest ancestors · 5

Same direction, newest first

2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
2023Let's Verify Step by Step31 May
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
2022Constitutional AI: Harmlessness from AI Feedback15 Dec
2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 50

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2023GPT-4 Technical Reportcited by 18,132
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2021Evaluating Large Language Models Trained on Codecited by 8,053
2020Measuring Massive Multitask Language Understandingcited by 7,768
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175

38 more

2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2020Learning to summarize from human feedbackcited by 2,562
2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2021Improving language models by retrieving from trillions of tokenscited by 1,278
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
2024Self-Rewarding Language Modelscited by 654
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
2022Self-critiquing models for assisting human evaluatorscited by 372
2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
2024Iterative Reasoning Preference Optimizationcited by 244
2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
2024Many-Shot In-Context Learningcited by 203
2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024LLM Critics Help Catch LLM Bugscited by 138
2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
2024Critique-out-Loud Reward Modelscited by 105
2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
2024Self-Taught Evaluatorscited by 70
2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
2024Improving Reward Models with Synthetic Critiquescited by 44
2024Transcendence: Generative Models Can Outperform The Experts That Train Themcited by 18
2024CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Taskscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 45; 2025: 271; 2026: 149.

465 papers.

Sort:

By area:

2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150

38 more

2025RM-R1: Reward Modeling as Reasoningcited by 149
2024Generative Reward Modelscited by 148
2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
2025Process Reward Models That Thinkcited by 118
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
2026Self-Distilled RLVRcited by 109
2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 95
2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
2025Self-rewarding correction for mathematical reasoningcited by 67
2025Can Large Reasoning Models Self-Train?cited by 66
2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
2024Aligning LLMs with Individual Preferences via Interactioncited by 63
2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61

50 most cited of 465 All 465 (JSON)

Similar, not cited

Close in wording, no citation either way