mlsanityData to 14 Oct 2026

A Unifying Lens on Reward Uncertainty in RLHF

arXiv 2606.09073 ↗First posted 8 Jun 2026cs.LG cs.AI cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural mitigation is pessimism: lowering rewards in regions where the RM is uncertain. However, standard scalar RMs provide no principled notion of uncertainty. We argue that the right object is a distributional reward model p(rmid x,y). Under either a Bayesian inference or a KL-distributionally robust optimization (KL-DRO) lens, the KL-regularized RLHF objective admits a closed-form effective reward tilde r(x,y) = ±βlogmathbbEp[e± r/β]. The pessimistic branch unifies the prior heuristics for RM ensemble aggregation: mean aggregation, worst-case optimization (WCO), and uncertainty-weighted optimization (UWO) all emerge as limits or truncations of this single expression. This also clarifies the implicit assumptions of each existing rule.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 48 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

35 of 48 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models27 Feb
  2. 2026Automatically Finding Reward Model Biases16 Feb
  3. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertain23 Oct
  4. 2025Mitigating Preference Hacking in Policy Optimization with Pessimism10 Mar
  5. 2024Reward-Robust RLHF in LLMs18 Sep
  6. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  7. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  8. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  9. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  10. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  12. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
3 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  12. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
8 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  4. 2022Defining and Characterizing Reward Hackingcited by 203
  5. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  6. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  7. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  8. 2024Efficient Exploration for LLMscited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Does Randomized Oversight Align AI Agents That Can Conceal?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  2. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  3. 2025Learning a Pessimistic Reward Model in RLHF
  4. 2026Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking
  5. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
  6. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
  7. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF