mlsanityData to 7 Oct 2026

Reward-Robust RLHF in LLMs

arXiv 2409.15360 ↗First posted 18 Sep 2024cs.LG cs.AI cs.CLcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alignment methods introduces significant challenges due to the inherent instability and imperfections of Reward Models (RMs), which can lead to critical issues such as reward hacking and misalignment with human intentions. In this paper, we introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges, paving the way for more reliable and resilient learning in LLMs. Our approach introduces a novel optimization objective that carefully balances performance and robustness by incorporating Bayesian Reward Model Ensembles (BRME) to model the uncertainty set of reward functions. This allows the framework to integrate both nominal performance and minimum reward signals, ensuring more stable learning even with imperfect RMs. Empirical results demonstrate that our framework consistently outperforms baselines across diverse benchmarks, showing improved accuracy and long-term stability. We also provide a theoretical analysis, demonstrating that reward-robust RLHF approaches the stability of constant reward settings, which proves to be acceptable even in a stochastic-case analysis. Together, these contributions highlight the framework potential to enhance both the performance and stability of LLM alignment.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 62 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

45 of 62 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  2. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  3. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards12 Mar
  4. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  5. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  6. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  7. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  9. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
2 more
  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2017Decoupled Weight Decay Regularizationcited by 1,722
  12. 2016Concrete Problems in AI Safetycited by 1,582
19 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  4. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2017Robust Adversarial Reinforcement Learningcited by 439
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2023Reward Design with Language Modelscited by 257
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  11. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  12. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  13. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 101
  14. 2018Reinforcement Learning with Perturbed Rewardscited by 76
  15. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  16. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  17. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  18. 2020Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertaintycited by 5
  19. 2024Exploring the LLM Journey from Cognition to Expression with Linear Representationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 16; 2026: 14.

32 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  5. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  6. 2025Adversarial Training of Reward Modelscited by 17
  7. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  8. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  9. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  10. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  11. 2024Baichuan4-Finance Technical Reportcited by 8
  12. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 6
20 more
  1. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5
  2. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  3. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  4. 2025Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interactioncited by 3
  5. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  6. 2026PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMscited by 3
  7. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  8. 2026Quantifying Empirical Compute-Supervision Tradeoffs in RLVRcited by 2
  9. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  10. 2025Probabilistic Uncertain Reward Modelcited by 1
  11. 2025Ethics and Persuasion in Reinforcement Learning from Human Feedback: A Procedural Rhetorical Approachcited by 1
  12. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  13. 2026Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferencescited by 1
  14. 2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
  15. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Rewardcited by 0
  16. 2026Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewardscited by 0
  17. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  18. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 0
  19. 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoningcited by 0
  20. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  5. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertain
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity
  7. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  8. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment