mlsanityData to 14 Oct 2026

Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

arXiv 2412.10616 ↗First posted 13 Dec 2024cs.LGcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy. On the other hand, while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart. We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 50 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

40 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 32

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  9. 2023Nash Learning from Human Feedbackcited by 224
  10. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  12. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
20 more
  1. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  3. 2023Is RLHF More Difficult than Standard RL?cited by 96
  4. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  5. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  7. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  8. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  9. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  10. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  11. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
  12. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  13. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  14. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 37
  15. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  16. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  17. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  18. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  19. 2024Harnessing Density Ratios for Online Reinforcement Learningcited by 15
  20. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6
  2. 2025Improved Bounds for Private and Robust Alignmentcited by 5
  3. 2026Cold-Start Personalization via Training-Free Priors from Structured World Modelscited by 3
  4. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  6. 2026Efficient Exploration at Scalecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration
  2. 2025Online Preference Alignment for Language Models via Count-based Exploration
  3. 2024Accelerated Preference Optimization for Large Language Model Alignment
  4. 2024OPTune: Efficient Online Preference Tuning
  5. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  7. 2024Investigating on RLHF methodology
  8. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences