mlsanityData to 7 Oct 2026

Sample Efficient Preference Alignment in LLMs via Active Exploration

arXiv 2312.00267 ↗First posted 1 Dec 2023cs.LG cs.AI stat.MLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference-based feedback is important for many applications in machine learning where evaluation of a reward function is not feasible. Notable recent examples arise in preference alignment for large language models, including in reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). For many applications of preference alignment, the cost of acquiring human feedback can be substantial. In this work, we take advantage of the fact that one can often choose contexts at which to obtain human feedback to most efficiently identify a good policy, and formalize the setting as an active contextual dueling bandit problem. We propose an active exploration algorithm to efficiently select the data and provide theoretical proof that it has a polynomial worst-case regret bound. We extend the setting and methodology for practical use in preference alignment of large language models. We provide two extensions, an online and an offline approach. Our method outperforms the baselines with limited samples of human preferences on several language models and four real-world datasets including two new datasets that we contribute to the literature.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 44 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

24 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 423
11 more
  1. 2017On Kernelized Multi-armed Banditscited by 348
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  3. 2015Contextual Dueling Banditscited by 101
  4. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  5. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  6. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  7. 2016Multi-fidelity Gaussian Process Bandit Optimisationcited by 31
  8. 2019Finding Generalizable Evidence by Learning to Convince Q&A Modelscited by 25
  9. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  10. 2019Zeroth Order Non-convex optimization with Dueling-Choice Banditscited by 11
  11. 2022Near-optimal Policy Identification in Active Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 3; 2026: 6.

18 papers.

Sort:

By area:

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  2. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  3. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  4. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  5. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  6. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
  7. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 18
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  9. 2025Active Learning for Direct Preference Optimizationcited by 9
  10. 2024Active Preference Learning for Ordering Items In- and Out-of-samplecited by 6
  11. 2024Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfecited by 5
  12. 2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Modelscited by 4
6 more
  1. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  2. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  3. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learningcited by 1
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  5. 2026Efficient Exploration at Scalecited by 0
  6. 2026A Finite Time Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning from Human Feedback with Active Queries
  2. 2024Active Preference Optimization for Sample Efficient RLHF
  3. 2025Online Preference Alignment for Language Models via Count-based Exploration
  4. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  5. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  6. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  7. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration
  8. 2024Active Preference Learning for Large Language Models