mlsanityData to 7 Oct 2026

RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style

arXiv 2410.16184 ↗First posted 21 Oct 2024cs.CLcited by 168

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are critical in techniques like Reinforcement Learning from Human Feedback (RLHF) and Inference Scaling Laws, where they guide language model alignment and select optimal responses. Despite their importance, existing reward model benchmarks often evaluate models by asking them to distinguish between responses generated by models of varying power. However, this approach fails to assess reward models on subtle but critical content changes and variations in style, resulting in a low correlation with policy model performance. To this end, we introduce RM-Bench, a novel benchmark designed to evaluate reward models based on their sensitivity to subtle content differences and resistance to style biases. Extensive experiments demonstrate that RM-Bench strongly correlates with policy model performance, making it a reliable reference for selecting reward models to align language models effectively. We evaluate nearly 40 reward models on RM-Bench. Our results reveal that even state-of-the-art models achieve an average performance of only 46.6%, which falls short of random-level accuracy (50%) when faced with style bias interference. These findings highlight the significant room for improvement in current reward models. Related code and data are available at https://github.com/THU-KEG/RM-Bench.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 49 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

38 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
18 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  13. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  14. 2024Nemotron-4 340B Technical Reportcited by 143
  15. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  16. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  17. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  18. 2024Inference-Time Language Model Alignment via Integrated Value Guidancecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 83; 2026: 81.

168 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  4. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  5. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  6. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  8. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  9. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  10. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  11. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  12. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
38 more
  1. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  4. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  7. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  8. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  11. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  12. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  13. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 29
  14. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  15. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  16. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  17. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  18. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewardscited by 25
  19. 2025Language Models that Think, Chat Bettercited by 23
  20. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  21. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  22. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  23. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  24. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  25. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19
  26. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  27. 2025Interpretable Reward Model via Sparse Autoencodercited by 17
  28. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 16
  29. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  30. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 16
  31. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  32. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  33. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  34. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  35. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 14
  36. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  37. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  38. 2025WorldPM: Scaling Human Preference Modelingcited by 12

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  2. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  3. 2024How to Evaluate Reward Models for RLHF
  4. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  5. 2024Improving Reward Models with Synthetic Critiques
  6. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  7. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  8. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?