mlsanityData to 7 Oct 2026

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

arXiv 2405.19332 ↗First posted 29 May 2024cs.LG cs.AIcited by 63

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike offline alignment with a fixed dataset, online feedback collection from humans or AI on model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving a globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, we propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. By solving the inner-level problem with the reparameterized reward function, the resulting algorithm, named Self-Exploring Language Models (SELM), eliminates the need for a separate RM and iteratively updates the LLM with a straightforward objective. Compared to Direct Preference Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen extrapolations and enhances exploration efficiency. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, SELM significantly boosts the performance on instruction-following benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard academic benchmarks in different settings. Our code and models are available at https://github.com/shenao-zhang/SELM.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 89 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

66 of 89 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  4. 2024sDPO: Don't Use Your Data All at Once28 Mar
  5. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  8. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  9. 2023Language Model Alignment with Elastic Reset6 Dec
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
3 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Training language models to follow instructions with human feedback4 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 51

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
38 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2024Self-Rewarding Language Modelscited by 654
  6. 2023Textbooks Are All You Needcited by 561
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  13. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  14. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  15. 2024Iterative Reasoning Preference Optimizationcited by 244
  16. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  17. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  18. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  19. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  20. 2023Self-Alignment with Instruction Backtranslationcited by 149
  21. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  22. 2021Epistemic Neural Networkscited by 120
  23. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  24. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  25. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  26. 2023Is RLHF More Difficult than Standard RL?cited by 96
  27. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  28. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  29. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  30. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  31. 2017Ensemble Samplingcited by 62
  32. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  33. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  34. 2024Efficient Exploration for LLMscited by 44
  35. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  36. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  37. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
  38. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 32; 2026: 13.

63 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  4. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  5. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  6. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  7. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  8. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  9. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  10. 2025A Survey of Direct Preference Optimizationcited by 35
  11. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  12. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
38 more
  1. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  3. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 18
  4. 2024Sample-Efficient Alignment for LLMscited by 16
  5. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
  6. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  7. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  8. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  9. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  10. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  11. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  12. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  13. 2024DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMscited by 9
  14. 2025Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewardscited by 9
  15. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Casescited by 8
  16. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  17. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  18. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  19. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  20. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  21. 2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
  22. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5
  23. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  24. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  25. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  26. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  27. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  28. 2025Mutual-Taught for Co-adapting Policy and Reward Modelscited by 3
  29. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  30. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  31. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  32. 2026Cold-Start Personalization via Training-Free Priors from Structured World Modelscited by 2
  33. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  34. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimizationcited by 1
  35. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?cited by 1
  36. 2025SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferencescited by 1
  37. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  38. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  2. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  3. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  4. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  7. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  8. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models