mlsanityData to 7 Oct 2026

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective

arXiv 2502.19255 ↗First posted 26 Feb 2025cs.LG cs.AI stat.MLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Sample efficiency is critical for online Reinforcement Learning from Human Feedback (RLHF). While existing works investigate sample-efficient online exploration strategies, the potential of utilizing misspecified yet relevant reward models to accelerate learning remains underexplored. This paper studies how to transfer knowledge from those imperfect reward models in online RLHF. We start by identifying a novel property due to KL-regularization in the RLHF objective: a policy's coverability of the optimal policy is captured by its sub-optimality. Building on this insight, we propose novel transfer learning principles and a theoretical algorithm -- emphTransfer Policy Optimization (TPO) -- with provable benefits compared to standard online learning. Empirically, inspired by our theoretical findings, we develop a win-rate-based transfer policy selection strategy with improved computational efficiency. Moreover, our empirical transfer learning technique is modular and can be integrated with various policy optimization methods, such as DPO, IPO and XPO, to further enhance their performance. We validate the effectiveness of our method through experiments on summarization tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 79 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

59 of 79 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHF7 Nov
  2. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignment23 Oct
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  4. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  5. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  6. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  7. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment18 Apr
  8. 2024Dataset Reset Policy Optimization for RLHF12 Apr
  9. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  10. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  11. 2024Self-Rewarding Language Models18 Jan
  12. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
7 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  2. 2023Nash Learning from Human Feedback1 Dec
  3. 2023Is RLHF More Difficult than Standard RL?25 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  10. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  11. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
28 more
  1. 2019A Theory of Regularized Markov Decision Processescited by 279
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  3. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  4. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  5. 2023AI Alignment: A Comprehensive Surveycited by 256
  6. 2024Iterative Reasoning Preference Optimizationcited by 244
  7. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  8. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  9. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  10. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  11. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  12. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  13. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  14. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  15. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  16. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  17. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  18. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  19. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  20. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  21. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  22. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  23. 2023Fast Rates for Maximum Entropy Explorationcited by 32
  24. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
  25. 2021Can Q-Learning be Improved with Advice?cited by 13
  26. 2024Scalable Online Exploration via Coverabilitycited by 10
  27. 2022Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regretcited by 3
  28. 2023Robust Knowledge Transfer in Tiered Reinforcement Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  2. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  3. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  4. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  3. 2025Reinforcement Learning from Human Feedback
  4. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  5. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  6. 2025Greedy Sampling Is Provably Efficient for RLHF
  7. 2025Reward Model Overoptimisation in Iterated RLHF
  8. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference