mlsanityData to 14 Oct 2026

Establishing Reliability Metrics for Reward Models in Large Language Models

arXiv 2504.14838 ↗First posted 21 Apr 2025cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The reward model (RM) that represents human preferences plays a crucial role in optimizing the outputs of large language models (LLMs), e.g., through reinforcement learning from human feedback (RLHF) or rejection sampling. However, a long challenge for RM is its uncertain reliability, i.e., LLM outputs with higher rewards may not align with actual human preferences. Currently, there is a lack of a convincing metric to quantify the reliability of RMs. To bridge this gap, we propose the textitunderlineReliable at underlineη (RETA) metric, which directly measures the reliability of an RM by evaluating the average quality (scored by an oracle) of the top η quantile responses assessed by an RM. On top of RETA, we present an integrated benchmarking pipeline that allows anyone to evaluate their own RM without incurring additional Oracle labeling costs. Extensive experimental studies demonstrate the superior stability of RETA metric, providing solid evaluations of the reliability of various publicly available and proprietary RMs. When dealing with an unreliable RM, we can use the RETA metric to identify the optimal quantile from which to select the responses.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 40 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

27 of 40 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  5. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  9. 2023On The Fragility of Learned Reward Functions9 Jan
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  12. 2022X-Risk Analysis for AI Researchcited by 60
2 more
  1. 2016Monte Carlo Markov Chain Algorithms for Sampling Strongly Rayleigh Distributions and Determinantal Point Processescited by 24
  2. 2015Efficient Sampling for k-Determinantal Point Processescited by 11

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34

Similar, not cited

Close in wording, no citation either way

  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  2. 2024How to Evaluate Reward Models for RLHF
  3. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  4. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  5. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  6. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  7. 2025Reward Models are Metrics in a Trench Coat
  8. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios