Data to 7 Oct 2026

Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles

arXiv 2401.00243 ↗First posted 30 Dec 2023cs.LGcited by 55

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) emerges as a promising paradigm for aligning large language models (LLMs). However, a notable challenge in RLHF is overoptimization, where beyond a certain threshold, the pursuit of higher rewards leads to a decline in human preferences. In this paper, we observe the weakness of KL regularization which is commonly employed in existing RLHF methods to address overoptimization. To mitigate this limitation, we scrutinize the RLHF objective in the offline dataset and propose uncertainty-penalized RLHF (UP-RLHF), which incorporates uncertainty regularization during RL-finetuning. To enhance the uncertainty quantification abilities for reward models, we first propose a diverse low-rank adaptation (LoRA) ensemble by maximizing the nuclear norm of LoRA matrix concatenations. Then we optimize policy models utilizing penalized rewards, determined by both rewards and uncertainties provided by the diverse reward LoRA ensembles. Our experimental results, based on two real human preference datasets, showcase the effectiveness of diverse reward LoRA ensembles in quantifying reward uncertainty. Additionally, uncertainty regularization in UP-RLHF proves to be pivotal in mitigating overoptimization, thereby contributing to the overall performance.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 54 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

43 of 54 references.

Closest ancestors · 14

Same direction, newest first

2 more

2022Uncertainty Estimation for Language Reward Models14 Mar
2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 29

2020Language Models are Few-Shot Learnerscited by 32,199
2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
2020Learning to summarize from human feedbackcited by 2,562
2015Trust Region Policy Optimizationcited by 2,503
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2022Red Teaming Language Models with Language Modelscited by 983
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906

17 more

2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2021A Survey of Uncertainty in Deep Neural Networkscited by 675
2023Baichuan 2: Open Large-scale Language Modelscited by 664
2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
2023Preference Ranking Optimization for Human Alignmentcited by 283
2019Self-Supervised Exploration via Disagreementcited by 263
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
2023LoRA ensembles for large language model fine-tuningcited by 65
2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 14; 2026: 15.

55 papers.

Sort:

By area:

2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
2024Robust Preference Optimization through Reward Model Distillationcited by 67
2024A Survey on LoRA of Large Language Modelscited by 67
2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 35
2024Bayesian Reward Models for LLM Alignmentcited by 35
2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
2024Reward-Robust RLHF in LLMscited by 32
2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29

38 more

2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
2024Language Models in Dialogue: Conversational Maxims for Human-AI Interactionscited by 21
2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
2025Adversarial Training of Reward Modelscited by 17
2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modelingcited by 15
2024Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned Modelscited by 15
2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
2026Real-Time Aligned Reward Model beyond Semanticscited by 12
2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
2024Towards Reliable Alignment: Uncertainty-aware RLHFcited by 7
2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
2024Uncertainty-Penalized Direct Preference Optimizationcited by 6
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
2025Energy-Based Reward Models for Robust Language Model Alignmentcited by 4
2025Bayesian Mixture of Experts For Large Language Modelscited by 4
2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
2024Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projectioncited by 2
2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
2025Policy Optimized Text-to-Image Pipeline Designcited by 1
2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
2024Online Self-Preferring Language Modelscited by 0
2025Towards Reliable, Uncertainty-Aware Alignmentcited by 0
2025The Silent Scholar Problem: A Probabilistic Framework for Breaking Epistemic Asymmetry in LLM Agentscited by 0
2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
2026Epistemic Uncertainty for Test-Time Discoverycited by 0
2026Reinforcement learning to improve large language model-based automated code compliance systemscited by 0
2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillationcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way