mlsanityData to 7 Oct 2026

Nearly Optimal Active Preference Learning and Its Application to LLM Alignment

arXiv 2602.01581 ↗First posted 2 Feb 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We evaluate our algorithm on real-world preference datasets and observe improved sample efficiency compared to existing methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 33 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

26 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Active Learning for Direct Preference Optimization3 Mar
  2. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  3. 2024Active Preference Learning for Large Language Models12 Feb

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 280
  6. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  7. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  8. 2014Best-Arm Identification in Linear Banditscited by 98
  9. 2019Sequential Experimental Design for Transductive Linear Banditscited by 88
  10. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  11. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  12. 2024Efficient Exploration for LLMscited by 44
11 more
  1. 2024Efficient Prompt Optimization Through the Lens of Best Arm Identificationcited by 35
  2. 2021Minimax Optimal Fixed-Budget Best Arm Identification in Linear Banditscited by 32
  3. 2020Improved Confidence Bounds for the Linear Logistic Model and Applications to Linear Banditscited by 31
  4. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  5. 2021Convergence of Uncertainty Sampling for Active Learningcited by 19
  6. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
  7. 2024Optimal Design for Human Preference Elicitationcited by 17
  8. 2018Uncertainty Sampling is Preconditioned Stochastic Gradient Descent on Zero-One Losscited by 10
  9. 2025Provably Efficient Online RLHF with One-Pass Reward Modelingcited by 8
  10. 2024Adaptive Experimentation When You Can't Experimentcited by 5
  11. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  4. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  5. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  6. 2026Optimal Design for Active Preference Learning with Biased LLM Judges
  7. 2024Sample-Efficient Alignment for LLMs
  8. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization