mlsanityData to 14 Oct 2026

Uncertainty-Aware Reward Modeling for Stable RLHF

arXiv 2606.19818 ↗First posted 18 Jun 2026cs.LG cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1) reward models cannot signal when their predictions are unreliable, since they usually act as deterministic point estimators; and (2) modern group-based policy optimization can amplify unreliable reward signals, as exemplified by GRPO's uniform treatment of rewards during advantage computation. As policies explore increasingly diverse responses, these two limitations create a critical vulnerability: unreliable reward estimates may be granted disproportionate influence, triggering severe reward hacking. We propose Uncertainty-Aware Reward Modeling (UARM), which equips reward models with calibrated uncertainty via quantile-based conformal prediction and reweights GRPO advantages through heteroscedastic variance decomposition. Experiments across HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

33 of 40 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks19 Mar
  2. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  3. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  6. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  8. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2025Group Sequence Policy Optimizationcited by 835
  9. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  10. 2019Conformalized Quantile Regressioncited by 629
  11. 2019Conformal Prediction Under Covariate Shiftcited by 519
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
11 more
  1. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  2. 2019Deep Evidential Regressioncited by 355
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  4. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  5. 2024Variational Bayesian Last Layerscited by 69
  6. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  7. 2024Provably Robust Conformal Prediction with Improved Efficiencycited by 27
  8. 2024Self-Calibrating Conformal Predictioncited by 16
  9. 2025CLEAR: Calibrated Learning for Epistemic and Aleatoric Riskcited by 6
  10. 2025Colorful Pinball: Density-Weighted Quantile Regression for Conditional Guarantee of Conformal Predictioncited by 3
  11. 2025Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flowscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidencecited by 2
  2. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  3. 2026S3: Spectral Null-Space Swap Makes Reasoning Models Efficientcited by 0
  4. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervalscited by 0
  5. 2026Finding Blind Spots in AppWorld and WorkArena Task Verifierscited by 0
  6. 2026Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  2. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  3. 2026A Unifying Lens on Reward Uncertainty in RLHF
  4. 2026Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking
  5. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  6. 2025Uncertainty-aware Reward Design Process
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown