mlsanityData to 7 Oct 2026

RewardBench 2: Advancing Reward Model Evaluation

arXiv 2506.01937 ↗First posted 2 Jun 2025cs.CLcited by 136

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are used throughout the post-training of language models to capture nuanced signals from preference data and provide a training target for optimization across instruction following, reasoning, safety, and more domains. The community has begun establishing best practices for evaluating reward models, from the development of benchmarks that test capabilities in specific skill areas to others that test agreement with human preferences. At the same time, progress in evaluation has not been mirrored by the effectiveness of reward models in downstream tasks -- simpler direct alignment algorithms are reported to work better in many cases. This paper introduces RewardBench 2, a new multi-skill reward modeling benchmark designed to bring new, challenging data for accuracy-based reward model evaluation -- models score about 20 points on average lower on RewardBench 2 compared to the first RewardBench -- while being highly correlated with downstream performance. Compared to most other benchmarks, RewardBench 2 sources new human prompts instead of existing prompts from downstream evaluations, facilitating more rigorous evaluation practices. In this paper, we describe our benchmark construction process and report how existing models perform on it, while quantifying how performance on the benchmark correlates with downstream use of the models in both inference-time scaling algorithms, like best-of-N sampling, and RLHF training algorithms like proximal policy optimization.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 76 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

64 of 76 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Reinforcement Learning from Human Feedback16 Apr
  2. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  3. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  4. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  5. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  8. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings20 Oct
  9. 2024How to Evaluate Reward Models for RLHF18 Oct
  10. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  11. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?8 Oct
  12. 2024Generative Reward Models2 Oct
11 more
  1. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  2. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  3. 2024Critique-out-Loud Reward Models21 Aug
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  6. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  7. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  8. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
29 more
  1. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2024A Survey on Data Selection for Language Modelscited by 230
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  12. 2024Nemotron-4 340B Technical Reportcited by 143
  13. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  14. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  15. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  16. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  17. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  18. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  19. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  20. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  21. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  22. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  23. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  24. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  25. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21
  26. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Modelscited by 21
  27. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  28. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  29. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 42; 2026: 94.

136 papers.

Sort:

By area:

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  2. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  3. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  4. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  5. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  6. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  7. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  8. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  9. 2025Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learningcited by 22
  10. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  11. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
  12. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 16
38 more
  1. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  2. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 16
  3. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
  4. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  5. 2025Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuitscited by 12
  6. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 12
  7. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  8. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  9. 2025SPARK: Synergistic Policy And Reward Co-Evolving Frameworkcited by 9
  10. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  11. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 8
  12. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  13. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 7
  14. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  15. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  16. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modelingcited by 7
  17. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  18. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 6
  19. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  20. 2025BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Executioncited by 5
  21. 2026PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Modelscited by 5
  22. 2026Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluationcited by 5
  23. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  24. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  25. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  26. 2025Continual Learning, Not Training: Online Adaptation For Agentscited by 4
  27. 2025Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statisticcited by 4
  28. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  29. 2026nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Spacecited by 4
  30. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 4
  31. 2025Reward Model Routing in Alignmentcited by 3
  32. 2025Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimalitycited by 3
  33. 2025Why DPO is a Misspecified Estimator and How to Fix Itcited by 3
  34. 2026Reward Models Inherit Value Biases from Pretrainingcited by 3
  35. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  36. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 3
  37. 2026Automatically Finding Reward Model Biasescited by 3
  38. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  2. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  3. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  4. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  5. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  6. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  7. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  8. 2023Tool-Augmented Reward Modeling