mlsanityData to 7 Oct 2026

RewardBench: Evaluating Reward Models for Language Modeling

arXiv 2403.13787 ↗First posted 20 Mar 2024cs.LGcited by 480

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are at the crux of successfully using RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. Resources for reward model training and understanding are sparse in the nascent open-source community around them. To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation. The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We create specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO). We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.

Key facts. Cited by 480 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 73 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

51 of 73 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2024Self-Rewarding Language Models18 Jan
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  5. 2023Interpreting Learned Feedback Patterns in Large Language Models12 Oct
  6. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  8. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  9. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
5 more
  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
22 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  10. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  11. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  12. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  13. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  14. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  15. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  16. 2023Generative Judge for Evaluating Alignmentcited by 163
  17. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  18. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  19. 2024Stable LM 2 1.6B Technical Reportcited by 86
  20. 2023Moral Foundations of Large Language Modelscited by 84
  21. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  22. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 139; 2025: 201; 2026: 140.

480 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  5. 2024DeepSeek-V3 Technical Reportcited by 724
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  7. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  9. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  11. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
38 more
  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  2. 2025Improving Video Generation with Human Feedbackcited by 267
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  4. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  7. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  8. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  10. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  11. 2024Nemotron-4 340B Technical Reportcited by 143
  12. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  13. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  14. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  15. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  16. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  17. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  18. 2025Process Reward Models That Thinkcited by 118
  19. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  20. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  21. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  22. 2024Critique-out-Loud Reward Modelscited by 105
  23. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  24. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  25. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  26. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  27. 2024How to Evaluate Reward Models for RLHFcited by 84
  28. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  29. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  30. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  31. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  32. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  33. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  34. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  35. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  36. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  37. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  38. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71

50 most cited of 480 All 480 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  2. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  3. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems
  4. 2026Automatically Finding Reward Model Biases
  5. 2026Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions
  6. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  7. 2024Transforming and Combining Rewards for Aligning Large Language Models
  8. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs