Data to 7 Oct 2026

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

arXiv 2410.00847 ↗First posted 1 Oct 2024cs.LGcited by 71

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are essential for aligning large language models (LLM) with human expectations. However, existing RMs struggle to capture the stochastic and uncertain nature of human preferences and fail to assess the reliability of reward predictions. To address these challenges, we introduce the Uncertainty-aware Reward Model (URM) and its ensemble variant, URME. URM employs a probabilistic value head to capture aleatoric uncertainty by modeling the distribution of disentangled human preference attributes. URME further quantifies epistemic uncertainty by examining discrepancies among individual URMs within the ensemble, enabling identification of unreliable evaluations. Our empirical evaluations demonstrate that URM achieves strong performance on RewardBench, outperforming competitive large-scale models. Additionally, extensive experiments, including best-of-n sampling (BoN), iterative direct preference optimization (iterative DPO), and proximal policy optimization (PPO), demonstrate that URM and URME significantly enhance LLMs' generation quality. Notably, reward predictions with lower uncertainty are far more reliable, demonstrate significantly higher quality, and result in substantially improved alignment.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 70 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

50 of 70 references.

Closest ancestors · 19

Same direction, newest first

7 more

Other references · 31

2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2020Learning to summarize from human feedbackcited by 2,562
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2022Large Language Models Encode Clinical Knowledgecited by 1,807
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748

19 more

2020MOPO: Model-based Offline Policy Optimizationcited by 630
2017Self-Normalizing Neural Networkscited by 591
2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
2018Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networkscited by 278
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2023Gender bias and stereotypes in Large Language Modelscited by 185
2024Nemotron-4 340B Technical Reportcited by 143
2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 113
2021Conservative Offline Distributional Reinforcement Learningcited by 78
2016On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Productscited by 52
2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
2024Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankerscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 36; 2026: 26.

71 papers.

Sort:

By area:

2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22

38 more

2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
2025Robust Reward Modeling via Causal Rubricscited by 18
2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedbackcited by 14
2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
2025Reward Model Overoptimisation in Iterated RLHFcited by 10
2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignmentcited by 8
2024Towards Reliable Alignment: Uncertainty-aware RLHFcited by 7
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
2026Reward Modeling from Natural Language Human Feedbackcited by 7
2025Data-adaptive Safety Rules for Training Reward Modelscited by 6
2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generationcited by 5
2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 5
2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUscited by 4
2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modelingcited by 4
2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulationcited by 4
2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoningcited by 3
2026Reward Models Inherit Value Biases from Pretrainingcited by 3
2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learningcited by 3
2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
2025Test-Time Scaling with Repeated Sampling Improves Multilingual Text Generationcited by 2
2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2
2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
2026SteerRM: Debiasing Reward Models via Sparse Autoencoderscited by 2
2026DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivitycited by 2
2025Probabilistic Uncertain Reward Modelcited by 1
2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregationcited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way