mlsanityData to 13 Oct 2026

Dual Active Learning for Reinforcement Learning from Human Feedback

arXiv 2410.02504 ↗First posted 3 Oct 2024stat.ML cs.LGcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. However, human feedback is costly and time-consuming, making it essential to collect high-quality conversation data for human teachers to label. Additionally, different human teachers have different levels of expertise. It is thus critical to query the most appropriate teacher for their opinions. In this paper, we use offline reinforcement learning (RL) to formulate the alignment problem. Motivated by the idea of D-optimal design, we first propose a dual active reward learning algorithm for the simultaneous selection of conversations and teachers. Next, we apply pessimistic RL to solve the alignment problem, based on the learned reward estimator. Theoretically, we show that the reward estimator obtained through our proposed adaptive selection strategy achieves minimal generalized variance asymptotically, and prove that the sub-optimality of our pessimistic policy scales as O(1/√T) with a given sample budget T. Through simulations and experiments on LLMs, we demonstrate the effectiveness of our algorithm and its superiority over state-of-the-arts.

Key facts. Cited by 19 papers in the map (data through 13 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 72 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

20 of 72 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback8 Mar
  2. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  3. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,112
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 908
  5. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  6. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  7. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 41
  8. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  9. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  10. 2024Synthetic Oversampling: Theory and A Practical Approach Using LLMs to Address Data Imbalancecited by 17
  11. 2023Enhancing Causal Reasoning in Large Language Models: A Causal Attribution Model for Precision Fine-Tuningcited by 14
  12. 2023Active teacher selection for reward learningcited by 7
4 more
  1. 2024Covariate Assisted Entity Ranking with Sparse Intrinsic Scorescited by 6
  2. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questionscited by 4
  3. 2024Recognizing Limits: Investigating Infeasibility in Large Language Modelscited by 4
  4. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 10; 2026: 9.

19 papers.

Sort:

By area:

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  2. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 15
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  4. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  5. 2025Active Learning for Direct Preference Optimizationcited by 10
  6. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedbackcited by 7
  7. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  9. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  10. 2025Active Measuring in Reinforcement Learning With Delayed Negative Effectscited by 1
  11. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  12. 2026Perturbation is All You Need for Extrapolating Language Modelscited by 1
7 more
  1. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  2. 2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0
  3. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  5. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignmentcited by 0
  6. 2026Learning Perturbations to Extrapolate Your LLMcited by 0
  7. 2026CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  2. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  4. 2024Active Preference Optimization for Sample Efficient RLHF
  5. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  6. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  8. 2024Optimal Design for Reward Modeling in RLHF