mlsanityData to 7 Oct 2026

Probabilistic Uncertain Reward Model

arXiv 2503.22480 ↗First posted 28 Mar 2025cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a critical technique for training large language models. However, conventional reward models based on the Bradley-Terry model (BTRM) often suffer from overconfidence when faced with inconsistent labels or out-of-distribution samples, leading to reward hacking, where the policy model blindly optimizes for proxy rewards while degrading true performance. This paper proposes the Probabilistic Uncertain Reward Model (PURM), which generalizes the Bradley-Terry model to learn the reward distributions that emerged from the preference data. We theoretically derive the loss function of PURM and introduce a novel method that uses the overlap between distributions to quantify uncertainty. Empirical results show that PURM outperforms existing methods with more accurate reward and sound uncertainty estimations, and sustains effective learning for more optimization steps and obtain higher maximum win rate in RLHF. The data and code of this paper are released at https://anonymous.4open.science/r/Probabilistic-Uncertain-Reward-Model/

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 27 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

17 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  2. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  3. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  4. 2024Reward-Robust RLHF in LLMs18 Sep
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec

Other references · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  9. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  10. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  2. 2026A Unifying Lens on Reward Uncertainty in RLHF
  3. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  4. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  5. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  6. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling
  7. 2025Learning a Pessimistic Reward Model in RLHF
  8. 2025On the Robustness of Reward Models for Language Model Alignment