mlsanityData to 7 Oct 2026

Active Learning for Direct Preference Optimization

arXiv 2503.01076 ↗First posted 3 Mar 2025cs.LG stat.MLcited by 9

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 54 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

32 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Active Preference Learning for Large Language Models12 Feb
  2. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  3. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration1 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  9. 2022Active Example Selection for In-Context Learningcited by 195
  10. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  11. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
  12. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
16 more
  1. 2015Cascading Bandits: Learning to Rank in the Cascade Modelcited by 75
  2. 2019Randomized Exploration in Generalized Linear Banditscited by 70
  3. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  4. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  5. 2016Cascading Bandits for Large-Scale Recommendation Problemscited by 45
  6. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  7. 2021Minimax Optimal Fixed-Budget Best Arm Identification in Linear Banditscited by 32
  8. 2016Multiple-Play Bandits in the Position-Based Modelcited by 27
  9. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  10. 2021Fixed-Budget Best-Arm Identification in Structured Banditscited by 22
  11. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
  12. 2024Optimal Design for Human Preference Elicitationcited by 17
  13. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  14. 2024Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Trainingcited by 6
  15. 2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learningcited by 5
  16. 2024Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  2. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 12
  3. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  4. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  5. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  6. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  7. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Rankingcited by 1
  8. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignmentcited by 0
  9. 2026Random Is Hard to Beat: Active Selection in online DPO with Modern LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024New Desiderata for Direct Preference Optimization
  2. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  4. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  5. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  6. 2024Minor DPO reject penalty to increase training robustness
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences
  8. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms