Data to 7 Oct 2026

Active Preference Learning for Large Language Models

arXiv 2402.08114 ↗First posted 12 Feb 2024cs.LG cs.AI cs.CLcited by 55

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

As large language models (LLMs) become more capable, fine-tuning techniques for aligning with human intent are increasingly important. A key consideration for aligning these models is how to most effectively use human resources, or model resources in the case where LLMs themselves are used as oracles. Reinforcement learning from Human or AI preferences (RLHF/RLAIF) is the most prominent example of such a technique, but is complex and often unstable. Direct Preference Optimization (DPO) has recently been proposed as a simpler and more stable alternative. In this work, we develop an active learning strategy for DPO to make better use of preference labels. We propose a practical acquisition function for prompt/completion pairs based on the predictive entropy of the language model and a measure of certainty of the implicit preference model optimized by DPO. We demonstrate how our approach improves both the rate of learning and final performance of fine-tuning on pairwise preference data.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 26 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

20 of 26 references.

Closest ancestors · 2

Same direction, newest first

Other references · 18

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2020Learning to summarize from human feedbackcited by 2,562
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2022Language Models (Mostly) Know What They Knowcited by 1,783
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
2017Deep Bayesian Active Learning with Image Datacited by 812
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672

6 more

2024Self-Rewarding Language Modelscited by 654
2020A Survey of Deep Active Learningcited by 547
2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 30; 2026: 14.

55 papers.

Sort:

By area:

2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
2024Active Preference Optimization for Sample Efficient RLHFcited by 61
2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
2024Sample-Efficient Alignment for LLMscited by 16
2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
2024Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinementcited by 13

38 more

2024Test-Time Alignment via Hypothesis Reweightingcited by 11
2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
2024WEPO: Web Element Preference Optimization for LLM-based Web Navigationcited by 10
2025Reward Model Overoptimisation in Iterated RLHFcited by 10
2025Active Learning for Direct Preference Optimizationcited by 9
2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 9
2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignmentcited by 8
2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6
2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5
2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUscited by 4
2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoningcited by 4
2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
2025On the Role of Preference Variance in Preference Optimizationcited by 3
2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
2026Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairingcited by 2
2026Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVRcited by 1
2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learningcited by 1
2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
2026Online Convex Optimization with Dueling Feedbackcited by 1
2025From Human Annotation to Automation: LLM-in-the-Loop Active Learning for Arabic Sentiment Analysiscited by 0
2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignmentcited by 0
2026Active Causal Experimentalist (ACE): Learning Intervention Strategies via Direct Preference Optimizationcited by 0
2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way