mlsanityData to 7 Oct 2026

Generative Reward Models

arXiv 2410.12832 ↗First posted 2 Oct 2024cs.LGcited by 148

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has greatly improved the performance of modern Large Language Models (LLMs). The RLHF process is resource-intensive and technically challenging, generally requiring a large collection of human preference labels over model-generated outputs. Reinforcement Learning from AI Feedback (RLAIF) addresses this data collection challenge by leveraging synthetic preferences generated by an LLM. However, recent work has shown that synthetic preferences labels may not align well with human preference judgments. To address this, we propose a hybrid approach that unifies RLHF and RLAIF methodologies. We introduce GenRM, an iterative algorithm that trains an LLM on self-generated reasoning traces, leading to synthetic preference labels matching human preference judgments. Empirically, we show that zero-shot LLM-based judgments under-perform compared to Bradley-Terry reward models on in-distribution tasks (between 9-36%). In contrast, GenRM achieves in-distribution accuracy comparable to Bradley-Terry models, while significantly outperforming them on out-of-distribution tasks (between 10-45%). Moreover, GenRM surpasses the performance of using LLMs as judges on both in-distribution (by 9-31%) and out-of-distribution tasks (by 2- 6%). Our results show that combining the strengths of RLHF and RLAIF offers a promising approach for improving the quality of synthetic preference labels.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 42 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

31 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  2. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
14 more
  1. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  3. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  5. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  6. 2024A Survey on Data Selection for Language Modelscited by 230
  7. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  8. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  9. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  10. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  11. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  12. 2023Specific versus General Principles for Constitutional AIcited by 41
  13. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 18
  14. 2023Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Datacited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 93; 2026: 50.

148 papers.

Sort:

By area:

  1. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2025Process Reward Models That Thinkcited by 118
  6. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  8. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  9. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  10. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  11. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  12. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
38 more
  1. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  2. 2025Self-rewarding correction for mathematical reasoningcited by 67
  3. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  4. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  5. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  6. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  7. 2024Self-Consistency Preference Optimizationcited by 50
  8. 2025Reward Reasoning Modelcited by 47
  9. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  10. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  11. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  12. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  13. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 39
  14. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  15. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  16. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  17. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  18. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 28
  19. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  20. 2025LLMs Can Generate a Better Answer by Aggregating Their Own Responsescited by 27
  21. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  22. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  23. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  24. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  25. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  26. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  27. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  28. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 19
  29. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  30. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  31. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  32. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  33. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 18
  34. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
  35. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 17
  36. 2025Learning to Generate Unit Tests for Automated Debuggingcited by 16
  37. 2025Incentivizing LLMs to Self-Verify Their Answerscited by 16
  38. 2025Dynamic and Generalizable Process Reward Modelingcited by 16

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  2. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  3. 2026ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  6. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  7. 2024Fine-Tuning Language Models with Reward Learning on Policy
  8. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling