mlsanityData to 7 Oct 2026

Reinforcement Learning from Human Feedback with Active Queries

arXiv 2402.09401 ↗First posted 14 Feb 2024cs.LG cs.AI cs.CLcited by 39

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled preference data, which is expensive to collect. In this paper, inspired by the success of active learning, we address this problem by proposing query-efficient RLHF methods. We first formalize the alignment problem as a contextual dueling bandit problem and design an active-query-based proximal policy optimization (APPO) algorithm with an tildeO(d2/Delta) instance-dependent regret bound and an tildeO(d2/Delta2) query complexity, where d is the dimension of feature space and Delta is the sub-optimality gap over all the contexts. We then propose ADPO, a practical version of our algorithm based on direct preference optimization (DPO) and apply it to fine-tuning LLMs. Our experiments show that ADPO, while only making about half of queries for human preference, matches the performance of the state-of-the-art DPO method.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 101 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

62 of 101 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Nash Learning from Human Feedback1 Dec
  2. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Is RLHF More Difficult than Standard RL?25 Jun
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2023Mistral 7Bcited by 3,065
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2018Is Q-learning Provably Efficient?cited by 485
  5. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  6. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  7. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  8. 2021Batch Active Learning at Scalecited by 127
  9. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  10. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 104
  11. 2015Contextual Dueling Banditscited by 101
  12. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  13. 2023Exploring the Feasibility of ChatGPT for Event Extractioncited by 76
  14. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  15. 2014Minimax Analysis of Active Learningcited by 61
  16. 2020Q-learning with Logarithmic Regretcited by 59
  17. 2022Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescited by 57
  18. 2024Active Preference Learning for Large Language Modelscited by 55
  19. 2022Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPscited by 52
  20. 2023Zero-shot Temporal Relation Extraction with ChatGPTcited by 52
  21. 2022Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptionscited by 51
  22. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  23. 2016Double Thompson Sampling for Dueling Banditscited by 42
  24. 2023Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiencycited by 39
  25. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 35
  26. 2015Sparse Dueling Banditscited by 29
  27. 2021Near-optimal Policy Optimization Algorithms for Learning Adversarial Linear Mixture MDPscited by 28
  28. 2023Variance-Aware Regret Bounds for Stochastic Contextual Dueling Banditscited by 22
  29. 2018Approximate Ranking from Pairwise Comparisonscited by 21
  30. 2020Active Preference Learning using Maximum Regretcited by 21
  31. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  32. 2017Maximum Selection and Ranking under Noisy Comparisonscited by 18
  33. 2023Borda Regret Minimization for Generalized Linear Dueling Banditscited by 17
  34. 2022Fast Rates in Pool-Based Batch Active Learningcited by 15
  35. 2018Active Reinforcement Learning with Monte-Carlo Tree Searchcited by 11
  36. 2019On Sample Complexity Upper and Lower Bounds for Exact Ranking from Noisy Comparisonscited by 10
  37. 2023On the Interplay Between Misspecification and Sub-optimality Gap in Linear Contextual Banditscited by 6
  38. 2021Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisonscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 19; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  2. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  3. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  4. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  7. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  8. 2025Language Model Personalization via Reward Factorizationcited by 27
  9. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  10. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
  11. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 18
  12. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
27 more
  1. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  2. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 11
  3. 2025Active Learning for Direct Preference Optimizationcited by 9
  4. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedbackcited by 7
  5. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  6. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  7. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5
  8. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Samplingcited by 4
  9. 2025Active Human Feedback Collection via Neural Contextual Dueling Banditscited by 4
  10. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  11. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  12. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  13. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  14. 2025Federated Linear Dueling Banditscited by 2
  15. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  16. 2025Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inferencecited by 2
  17. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  18. 2024The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedbackcited by 1
  19. 2024A Theoretical Survey on Foundation Modelscited by 1
  20. 2024FTSmartAudit: A Knowledge Distillation-Enhanced Framework for Automated Smart Contract Auditing Using Fine-Tuned LLMscited by 1
  21. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hackingcited by 1
  22. 2025Active Measuring in Reinforcement Learning With Delayed Negative Effectscited by 1
  23. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0
  24. 2026Efficient Exploration at Scalecited by 0
  25. 2026Efficiently Aligning Language Models with Online Natural Language Feedbackcited by 0
  26. 2026How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysiscited by 0
  27. 2026When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  4. 2026Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
  5. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  6. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
  7. 2025Online Preference Alignment for Language Models via Count-based Exploration
  8. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment