mlsanityData to 14 Oct 2026

Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

arXiv 2410.05584 ↗First posted 8 Oct 2024cs.LG cs.AI cs.CLcited by 31

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reward Models (RMs) are crucial for aligning language models with human preferences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored. In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance. Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization. This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 55 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

34 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Evaluating Robustness of Reward Models for Mathematical Reasoning2 Oct
  2. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
19 more
  1. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  2. 2022Defining and Characterizing Reward Hackingcited by 203
  3. 2018Categorizing Variants of Goodhart's Lawcited by 149
  4. 2021Consequences of Misaligned AIcited by 80
  5. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  6. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  7. 2020Quantifying Differences in Reward Functionscited by 63
  8. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  9. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  10. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  11. 2023Goodhart's Law in Reinforcement Learningcited by 41
  12. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  13. 2020Understanding Learned Reward Functionscited by 37
  14. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  15. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  16. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  17. 2024When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedbackcited by 17
  18. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  19. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 14.

31 papers.

Sort:

By area:

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  2. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  3. 2025RewardDance: Reward Scaling in Visual Generationcited by 68
  4. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 63
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  7. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 11
  8. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  10. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 8
  11. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  12. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
19 more
  1. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  2. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 6
  3. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  4. 2025Scalable Oversight for Superhuman AI via Recursive Self-Critiquingcited by 4
  5. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  6. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  7. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  8. 2025Reward Models are Metrics in a Trench Coatcited by 2
  9. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  10. 2026AI Alignment From Social Choice Perspectivescited by 2
  11. 2026Unifying Adversarial Robustness and Training Across Text Scoring Modelscited by 0
  12. 2026Safe Reinforcement Learning with Preference-based Constraint Inferencecited by 0
  13. 2026PARM: Pipeline-Adapted Reward Modelcited by 0
  14. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewardscited by 0
  15. 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
  16. 2026JET: Judge-Guided Evolution at Test Time for Agent Programscited by 0
  17. 2026TRACE: Diagnosing Verifier Brittleness in Agentic Evaluationcited by 0
  18. 2026Unsupervised Multimodal Reward Modelingcited by 0
  19. 2026Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gapcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  3. 2025Reward Models Identify Consistency, Not Causality
  4. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  5. 2026Reward Models Inherit Value Biases from Pretraining
  6. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  7. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
  8. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling