mlsanityData to 14 Oct 2026

Variance-aware Reward Modeling with Anchor Guidance

arXiv 2605.11865 ↗First posted 12 May 2026stat.ML cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Standard Bradley--Terry (BT) reward models are limited when human preferences are pluralistic. Although soft preference labels preserve disagreement information, BT can only express it by shrinking reward margins. Gaussian reward models provide an alternative by jointly predicting a reward mean and a reward variance, but suffer from a fundamental non-identifiability from pairwise preferences alone. We propose Anchor-guided Variance-aware Reward Modeling, a framework that resolves this non-identifiability by augmenting preference data with two coarse response-level anchor labels. Building on this, we prove that two anchors are sufficient for identification, develop a joint training objective and establish a non-asymptotic convergence rate for both the estimated reward mean and variance functions. Across simulation studies and four real-world diverging-preference datasets, our method consistently improves reward modeling performance and downstream RLHF, including PPO training and best-of-N selection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

38 of 50 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective2 Apr
  2. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback13 Mar
  3. 2026Learning Ordinal Probabilistic Reward from Preferences13 Feb
  4. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback2 Dec
  5. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementary10 Jul
  6. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages16 May
  7. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?26 Feb
  8. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  9. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd19 Nov
  10. 2024How to Evaluate Reward Models for RLHF18 Oct
  11. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  12. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
10 more
  1. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning19 Aug
  2. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  4. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  5. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  5. 2022On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networkscited by 120
  6. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  7. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  8. 2019Reliable training and estimation of variance networkscited by 63
  9. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
  10. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 54
  11. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  12. 2022Faithful Heteroscedastic Regression with Neural Networkscited by 37
4 more
  1. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  2. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 26
  3. 2025An Overview of Large Language Models for Statisticianscited by 24
  4. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Debiasing Reward Models by Representation Learning with Guarantees
  2. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
  3. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  4. 2026Bayesian Preference Learning for Test-Time Steerable Reward Models
  5. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles
  6. 2024Reward Learning From Preference With Ties
  7. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  8. 2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport