# A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

arXiv: https://arxiv.org/abs/2311.14743 · HTML: https://mlsanity.com/paper/2311.14743/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://mlsanity.com/paper/2209.09858/) (2022)
- [Training OOD Detectors in their Natural Habitats](https://mlsanity.com/paper/2202.03299/) (2022)
- [Diagnosing Model Performance Under Distribution Shift](https://mlsanity.com/paper/2303.02011/) (2023)
- [Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models](https://mlsanity.com/paper/2303.12748/) (2023)
- [How Good Are LLMs at Out-of-Distribution Detection?](https://mlsanity.com/paper/2308.10261/) (2023)
- [Accurate Layerwise Interpretable Competence Estimation](https://mlsanity.com/paper/1910.11363/) (2019)
- [Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains](https://mlsanity.com/paper/2311.07723/) (2023)

## Built on by (most cited first, 18 of 18)

- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [Filtered Direct Preference Optimization](https://mlsanity.com/paper/2404.13846/) (2024)
- [Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison](https://mlsanity.com/paper/2409.09603/) (2024)
- [Robust LLM Alignment via Distributionally Robust Direct Preference Optimization](https://mlsanity.com/paper/2502.01930/) (2025)
- [Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients](https://mlsanity.com/paper/2510.18924/) (2025)
- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [Jailbreaking as a Reward Misspecification Problem](https://mlsanity.com/paper/2406.14393/) (2024)
- [AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning](https://mlsanity.com/paper/2510.14738/) (2025)
- [Interpreting Language Reward Models via Contrastive Explanations](https://mlsanity.com/paper/2411.16502/) (2024)
- [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [MetaRM: Shifted Distributions Alignment via Meta-Learning](https://mlsanity.com/paper/2405.00438/) (2024)
- [Online Distributionally Robust LLM Alignment via Regression to Relative Reward](https://mlsanity.com/paper/2509.19104/) (2025)
- [BayesFlow: A Probability Inference Framework for Meta-Agent Assisted Workflow Generation](https://mlsanity.com/paper/2601.22305/) (2026)
- [Adaptive Multilevel Twisted Sequential Monte Carlo for Rare Events Estimation in Language Models](https://mlsanity.com/paper/2608.21736/) (2026)
- [Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy](https://mlsanity.com/paper/2401.12129/) (2024)
- [Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions](https://mlsanity.com/paper/2601.16987/) (2026)

## Abstract (arXiv)

Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.