mlsanityData to 7 Oct 2026

RMB: Comprehensively Benchmarking Reward Models in LLM Alignment

arXiv 2410.09893 ↗First posted 13 Oct 2024cs.CLcited by 60

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) guide the alignment of large language models (LLMs), steering them toward behaviors preferred by humans. Evaluating RMs is the key to better aligning LLMs. However, the current evaluation of RMs may not directly correspond to their alignment performance due to the limited distribution of evaluation data and evaluation methods that are not closely related to alignment objectives. To address these limitations, we propose RMB, a comprehensive RM benchmark that covers over 49 real-world scenarios and includes both pairwise and Best-of-N (BoN) evaluations to better reflect the effectiveness of RMs in guiding alignment optimization. We demonstrate a positive correlation between our benchmark and the downstream alignment task performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art RMs, revealing their generalization defects that were not discovered by previous benchmarks, and highlighting the potential of generative RMs. Furthermore, we delve into open questions in reward models, specifically examining the effectiveness of majority voting for the evaluation of reward models and analyzing the impact factors of generative RMs, including the influence of evaluation criteria and instructing methods. Our evaluation code and datasets are available at https://github.com/Zhou-Zoey/RMB-Reward-Model-Benchmark.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 52 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

36 of 52 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Critique-out-Loud Reward Models21 Aug
  2. 2024Reinforcement Learning for LLM Post-Training: A Survey23 Jul
  3. 2024BOND: Aligning LLMs with Best-of-N Distillation19 Jul
  4. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Training4 Jul
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  6. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024Self-Rewarding Language Models18 Jan
  9. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  10. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  11. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  12. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
3 more
  1. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning18 Oct
  2. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2022Large Language Models Can Self-Improvecited by 656
9 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  4. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  5. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  6. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  7. 2024Variational Best-of-N Alignmentcited by 49
  8. 2023RealBehavior: A Framework for Faithfully Characterizing Foundation Models' Human-like Behavior Mechanismscited by 5
  9. 2024SEAL: Systematic Error Analysis for Value ALignmentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 37; 2026: 19.

60 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  4. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  5. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  6. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  7. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  8. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  9. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  10. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  11. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  12. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
38 more
  1. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  2. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  3. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  4. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19
  5. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  6. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  7. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  8. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  9. 2025WorldPM: Scaling Human Preference Modelingcited by 12
  10. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 11
  11. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  12. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 10
  13. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 8
  14. 2025ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Modelscited by 8
  15. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 8
  16. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  17. 2025DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHFcited by 7
  18. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 7
  19. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  20. 2026WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agentscited by 7
  21. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  22. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  23. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  24. 2025URPO: A Unified Reward & Policy Optimization Framework for Large Language Modelscited by 5
  25. 2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
  26. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  27. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  28. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  29. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  30. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  31. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  32. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  33. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judgescited by 3
  34. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 3
  35. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
  36. 2025Reward Models are Metrics in a Trench Coatcited by 2
  37. 2026What Does Preference Learning Recover from Pairwise Comparison Data?cited by 2
  38. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  2. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  3. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  5. 2023On Diversified Preferences of Large Language Model Alignment
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  8. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation