mlsanityData to 14 Oct 2026

PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration

arXiv 2606.27578 ↗First posted 25 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models for Reinforcement Learning from Human Feedback (RLHF) pool preferences across thousands of annotators and fit one global affine calibrator, collapsing raters with systematically different rating-scale offsets and slopes into a single average-rater fit that does not match any individual annotator. PEBS is a per-rater empirical-Bayes shrinkage estimator: it fits per-rater affine calibrators on a held-out slice of each annotator's ratings and applies Morris-James-Stein empirical-Bayes shrinkage toward the population mean, in closed form and without retraining the reward model. On PRISM, PEBS reduces within-user held-out RMSE by 8.58% over the pooled population-slope baseline. The procedure replicates on PluriHarms harm ratings (Qwen-2.5 base, in-family) with a +9.66% RMSE reduction over the same population-slope baseline. PEBS is a closed-form post-hoc estimator for annotator-specific affine calibration in RLHF reward modeling; it leaves the reward base model unchanged and estimates only the rater-level map used at inference time for new ratings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 47 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

29 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback2 Dec
  2. 2025Preference Learning for AI Alignment: a Causal Perspective6 Jun
  3. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  6. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  7. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  8. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  9. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  10. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  11. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  12. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
7 more
  1. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 26
  2. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  3. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  4. 2026EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimizationcited by 5
  5. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  6. 2015Optimal shrinkage estimation in heteroscedastic hierarchical linear modelscited by 3
  7. 2025The Reward Model Selection Crisis in Personalized Alignmentcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rater State Bias in RLHF Preference Data: An Audit Framework
  2. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  3. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
  4. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  6. 2025Learning a Pessimistic Reward Model in RLHF
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning