mlsanityData to 7 Oct 2026

Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

arXiv 2602.10623 ↗First posted 11 Feb 2026cs.LG cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models learned from human preferences are central to aligning large language models (LLMs) via reinforcement learning from human feedback, yet they are often vulnerable to reward hacking due to noisy annotations and systematic biases such as response length or style. We propose Bayesian Non-Negative Reward Model (BNRM), a principled reward modeling framework that integrates non-negative factor analysis into Bradley-Terry (BT) preference model. BNRM represents rewards through a sparse, non-negative latent factor generative process that operates at two complementary levels: instance-specific latent variables induce disentangled reward representations, while sparsity over global latent factors acts as an implicit debiasing mechanism that suppresses spurious correlations. Together, this disentanglement-then-debiasing structure enables robust uncertainty-aware reward learning. To scale BNRM to modern LLMs, we develop an amortized variational inference network conditioned on deep model representations, allowing efficient end-to-end training. Extensive empirical results demonstrate that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 79 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

53 of 79 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance29 Dec
  2. 2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport13 Oct
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  6. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  7. 2024From Lists to Emojis: How Format Bias Affects Model Alignment18 Sep
  8. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  9. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  10. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  11. 2024Bayesian Reward Models for LLM Alignment20 Feb
  12. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
9 more
  1. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  4. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  5. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  8. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
20 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  3. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  4. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  5. 2023Instruction Tuning with GPT-4cited by 676
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  9. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  10. 2022On the Identifiability of Nonlinear ICA: Sparsity and Beyondcited by 84
  11. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  12. 2016Towards Bayesian Deep Learning: A Framework and Some Existing Methodscited by 57
  13. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  14. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  15. 2021Sawtooth Factorial Topic Embeddings Guided Gamma Belief Networkcited by 23
  16. 2022Chroma-VAE: Mitigating Shortcut Learning with Generative Classifierscited by 17
  17. 2015The Poisson Gamma Belief Networkcited by 10
  18. 2020Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inferencecited by 10
  19. 2024A Non-negative VAE:the Generalized Gamma Belief Networkcited by 4
  20. 2025Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  2. 2026Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuningcited by 2
  3. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  4. 2026ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interactioncited by 0
  5. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
  2. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
  3. 2026Factored Causal Representation Learning for Robust Reward Modeling in RLHF
  4. 2026Bayesian Preference Learning for Test-Time Steerable Reward Models
  5. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  6. 2025Debiasing Reward Models by Representation Learning with Guarantees
  7. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  8. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking