mlsanityData to 7 Oct 2026

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

arXiv 2311.14743 ↗First posted 21 Nov 2023cs.CL cs.LGcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 31 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

25 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains13 Nov
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  8. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  9. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
10 more
  1. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  2. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  3. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  4. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  5. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  6. 2022Training OOD Detectors in their Natural Habitatscited by 86
  7. 2023Diagnosing Model Performance Under Distribution Shiftcited by 37
  8. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Modelscited by 17
  9. 2023How Good Are LLMs at Out-of-Distribution Detection?cited by 15
  10. 2019Accurate Layerwise Interpretable Competence Estimationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 9; 2026: 2.

17 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  3. 2024Filtered Direct Preference Optimizationcited by 24
  4. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 20
  5. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  6. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  7. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  8. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  9. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  10. 2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
  11. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  12. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
5 more
  1. 2024MetaRM: Shifted Distributions Alignment via Meta-Learningcited by 4
  2. 2026BayesFlow: A Probability Inference Framework for Meta-Agent Assisted Workflow Generationcited by 2
  3. 2024Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energycited by 0
  4. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Rewardcited by 0
  5. 2026Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  3. 2024Bayesian Reward Models for LLM Alignment
  4. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models
  5. 2025Reward Models are Metrics in a Trench Coat
  6. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  7. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  8. 2025A Systematic Analysis of Base Model Choice for Reward Modeling