mlsanityData to 7 Oct 2026

How to Evaluate Reward Models for RLHF

arXiv 2410.14872 ↗First posted 18 Oct 2024cs.LG cs.AI cs.CLcited by 84

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback). The gold-standard approach is to run a full RLHF training pipeline and directly probe downstream LLM performance. However, this process is prohibitively expensive. To address this, we build a predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. These proxy tasks consist of a large-scale human preference and a verifiable correctness preference dataset, in which we measure 12 metrics across 12 domains. To investigate which reward model metrics are most correlated to gold-standard RLHF outcomes, we launch an end-to-end RLHF experiment on a large-scale crowdsourced human preference platform to view real reward model downstream performance as ground truth. Ultimately, we compile our data and findings into Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly linked to post-RLHF real-world human preference performance, which we open-source for public use and further development. Our code and evaluations can be found at https://github.com/lmarena/PPE .

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 32 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

20 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  12. 2024InternLM2 Technical Reportcited by 391
2 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 44; 2026: 39.

84 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  6. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  7. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  8. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  9. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  12. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
38 more
  1. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  2. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  3. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  4. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  5. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  6. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  7. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  8. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  9. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  10. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19
  11. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 19
  12. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  13. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  14. 2025Reliable Fine-Grained Evaluation of Natural Language Math Proofscited by 14
  15. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
  16. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  17. 2025WorldPM: Scaling Human Preference Modelingcited by 12
  18. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  19. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  20. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8
  21. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  22. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  23. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 7
  24. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  25. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  26. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  27. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  28. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  29. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  30. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  31. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 5
  32. 2025Improving Model Alignment Through Collective Intelligence of Open-Source LLMScited by 4
  33. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  34. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  35. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  36. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  37. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  38. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  5. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  6. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  7. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  8. 2025A Systematic Analysis of Base Model Choice for Reward Modeling