mlsanityData to 14 Oct 2026

Bayesian Reward Models for LLM Alignment

arXiv 2402.13210 ↗First posted 20 Feb 2024cs.LGcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

To ensure that large language model (LLM) responses are helpful and non-toxic, a reward model trained on human preference data is usually used. LLM responses with high rewards are then selected through best-of-n (BoN) sampling or the LLM is further optimized to produce responses with high rewards through reinforcement learning from human feedback (RLHF). However, these processes are susceptible to reward overoptimization or `hacking', where responses receive high rewards due to imperfections in the reward model rather than true preference, particularly as prompts or responses deviate from the training data. To address these challenges, we propose to train a Bayesian reward model, which signals higher uncertainty further from the training data distribution. We trained Bayesian reward models using Laplace approximation on LoRA weights, and found that the resulting uncertainty estimates can effectively mitigate reward overoptimization in BoN sampling.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 42 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

34 of 42 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  2. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  3. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  4. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2022Uncertainty Estimation for Language Reward Models14 Mar
  10. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  8. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  9. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  11. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  12. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
12 more
  1. 2021Bayesian Neural Network Priors Revisitedcited by 121
  2. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  3. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 80
  4. 2023LoRA ensembles for large language model fine-tuningcited by 67
  5. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  6. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  7. 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
  8. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  9. 2020Deep kernel processescited by 34
  10. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  11. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 28
  12. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 12; 2026: 12.

35 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  5. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 19
  6. 2024Bayesian Concept Bottleneck Models with LLM Priorscited by 17
  7. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  8. 2025Reward Model Overoptimisation in Iterated RLHFcited by 12
  9. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 12
  10. 2024Understanding the Role of User Profile in the Personalization of Large Language Modelscited by 11
  11. 2024A SMART Mnemonic Sounds like "Glue Tonic": Mixing LLMs with Student Feedback to Make Mnemonic Learning Stickcited by 10
  12. 2025Thinking Preference Optimizationcited by 8
23 more
  1. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 7
  2. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweightingcited by 6
  3. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 6
  4. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
  5. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
  6. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  7. 2024How Useful is Intermittent, Asynchronous Expert Feedback for Bayesian Optimization?cited by 3
  8. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  9. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  10. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  11. 2024Why you don't overfit, and don't need Bayes if you only train for one epochcited by 2
  12. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  13. 2025The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectivescited by 2
  14. 2026From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Modelscited by 2
  15. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 2
  16. 2024Solving the Inverse Alignment Problem for Efficient RLHFcited by 1
  17. 2024When Can Proxies Improve the Sample Complexity of Preference Learning?cited by 1
  18. 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoningcited by 1
  19. 2025LLM Reinforcement in Contextcited by 0
  20. 2025Invasive Context Engineering to Control Large Language Modelscited by 0
  21. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  22. 2026Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewardscited by 0
  23. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  2. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
  3. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  4. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  5. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  6. 2024Reward-Robust RLHF in LLMs
  7. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  8. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertain