mlsanityData to 14 Oct 2026

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

arXiv 2405.21046 ↗First posted 31 May 2024cs.LG cs.AI cs.CLcited by 106

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as a central tool for language model alignment. We consider online exploration in RLHF, which exploits interactive access to human or AI feedback by deliberately encouraging the model to produce diverse, maximally informative responses. By allowing RLHF to confidently stray from the pre-trained model, online exploration offers the possibility of novel, potentially super-human capabilities, but its full potential as a paradigm for language model training has yet to be realized, owing to computational and statistical bottlenecks in directly adapting existing reinforcement learning techniques. We propose a new algorithm for online exploration in RLHF, Exploratory Preference Optimization (XPO), which is simple and practical -- a one-line change to (online) Direct Preference Optimization (DPO; Rafailov et al., 2023) -- yet enjoys the strongest known provable guarantees and promising empirical performance. XPO augments the DPO objective with a novel and principled exploration bonus, empowering the algorithm to explore outside the support of the initial model and human feedback data. In theory, we show that XPO is provably sample-efficient and converges to a near-optimal language model policy under natural exploration conditions, irrespective of whether the initial model has good coverage. Our analysis, which builds on the observation that DPO implicitly performs a form of Qstar-approximation (or, Bellman error minimization), combines previously disparate techniques from language modeling and theoretical reinforcement learning in a serendipitous fashion through the perspective of KL-regularized Markov decision processes. Empirically, we find that XPO is more sample-efficient than non-exploratory DPO variants in a preliminary evaluation.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 73 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

55 of 73 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024REBEL: Reinforcement Learning via Regressing Relative Rewards25 Apr
  4. 2024Dataset Reset Policy Optimization for RLHF12 Apr
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  6. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  7. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  8. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  9. 2023Nash Learning from Human Feedback1 Dec
  10. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  11. 2023Is RLHF More Difficult than Standard RL?25 Jun
  12. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
4 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  12. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
27 more
  1. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  2. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  3. 2024Iterative Reasoning Preference Optimizationcited by 246
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  5. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  6. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  7. 2017A unified view of entropy-regularized Markov decision processescited by 182
  8. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  9. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  10. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  11. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  12. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  13. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  14. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  15. 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
  16. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  17. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  18. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  19. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  20. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
  21. 2024Efficient Exploration for LLMscited by 45
  22. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  23. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  24. 2023Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficientcited by 30
  25. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  26. 2022KL-Entropy-Regularized RL with a Generative Model is Minimax Optimalcited by 13
  27. 2024Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 51; 2026: 35.

106 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  2. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  4. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  5. 2024Process Reward Model with Q-Value Rankingscited by 80
  6. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  7. 2025Self-rewarding correction for mathematical reasoningcited by 70
  8. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  9. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  10. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  11. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  12. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
38 more
  1. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 29
  2. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  4. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
  5. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 26
  6. 2026On the Position Bias of On-Policy Distillationcited by 26
  7. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  9. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 23
  10. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  11. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  12. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  13. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 17
  14. 2024Sample-Efficient Alignment for LLMscited by 16
  15. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 16
  16. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  17. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
  18. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 13
  19. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  20. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  21. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  22. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
  23. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 11
  24. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 11
  25. 2026Reward-free Alignment for Conflicting Objectivescited by 11
  26. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  27. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  28. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  29. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10
  30. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  31. 2026Beyond Pessimism: Offline Learning in KL-regularized Gamescited by 10
  32. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  33. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  34. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  35. 2025Multiplayer Nash Preference Optimizationcited by 7
  36. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  37. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6
  38. 2025Rectifying Shortcut Behaviors in Preference-based Reward Learningcited by 6

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Accelerated Preference Optimization for Large Language Model Alignment
  5. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  6. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  7. 2024OPTune: Efficient Online Preference Tuning
  8. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration