mlsanityData to 7 Oct 2026

Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation

arXiv 2403.05171 ↗First posted 8 Mar 2024cs.LG cs.AIcited by 33

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We introduce Adversarial Policy Optimization (AdvPO), a novel solution to the pervasive issue of reward over-optimization in Reinforcement Learning from Human Feedback (RLHF) for Large Language Models (LLMs). Over-optimization occurs when a reward model serves as an imperfect proxy for human preference, and RL-driven policy optimization erroneously exploits reward inaccuracies. In this paper, we begin by introducing a lightweight way to quantify uncertainties in rewards, relying solely on the last layer embeddings of the reward model, without the need for computationally expensive reward ensembles. AdvPO then addresses a distributionally robust optimization problem centred around the confidence interval of the reward model's predictions for policy improvement. Through comprehensive experiments on the Anthropic HH and TL;DR summarization datasets, we illustrate the efficacy of AdvPO in mitigating the overoptimization issue, consequently resulting in enhanced performance as evaluated through human-assisted evaluation.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 49 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

37 of 49 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  2. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Data17 Dec
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
1 more
  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
12 more
  1. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  6. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  7. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  8. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  9. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  10. 2020Neural Contextual Bandits with Deep Representation and Shallow Explorationcited by 71
  11. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  12. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 14; 2026: 8.

33 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  3. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2024Reward-Robust RLHF in LLMscited by 32
  6. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  7. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  8. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  9. 2024Filtered Direct Preference Optimizationcited by 24
  10. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  11. 2024Scalable Ensembling For Mitigating Reward Overoptimisationcited by 14
  12. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
21 more
  1. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  2. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  3. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  4. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  5. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  6. 2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignmentcited by 8
  7. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  8. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
  9. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  10. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  11. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
  12. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignmentcited by 4
  13. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUscited by 4
  14. 2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
  15. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  16. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregationcited by 1
  17. 2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
  18. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Rewardcited by 0
  19. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  20. 2026Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guidecited by 0
  21. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  2. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
  3. 2025Mitigating Preference Hacking in Policy Optimization with Pessimism
  4. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  7. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
  8. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback