mlsanityData to 7 Oct 2026

Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference

arXiv 2511.04286 ↗First posted 6 Nov 2025cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human preferences is a cornerstone of aligning machine learning models with subjective human judgments. Yet, collecting such preference data is often costly and time-consuming, motivating the need for more efficient learning paradigms. Two established approaches offer complementary advantages: RLHF scales effectively to high-dimensional tasks such as LLM fine-tuning, while PBO achieves greater sample efficiency through active querying. We propose a hybrid framework that unifies RLHF's scalability with PBO's query efficiency by integrating an acquisition-driven module into the RLHF pipeline, thereby enabling active and sample-efficient preference gathering. We validate the proposed approach on two representative domains: (i) high-dimensional preference optimization and (ii) LLM fine-tuning. Experimental results demonstrate consistent improvements in both sample efficiency and overall performance across these tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 23 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

12 of 23 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards12 Mar
  2. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 6

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  4. 2017Preferential Bayesian Optimizationcited by 72
  5. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  6. 2018Active Reinforcement Learning with Monte-Carlo Tree Searchcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  2. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  5. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  6. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  7. 2024Active Preference Optimization for Sample Efficient RLHF
  8. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling