mlsanityData to 14 Oct 2026

UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning

arXiv 2606.19328 ↗First posted 17 Jun 2026cs.LG cs.AI cs.ROcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 35 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

28 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  2. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning12 Nov
  3. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  3. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  6. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  7. 2017On Kernelized Multi-armed Banditscited by 352
  8. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  9. 2020Model-Based Offline Planningcited by 129
  10. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  11. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 90
  12. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
12 more
  1. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  2. 2021On the Expressivity of Markov Rewardcited by 71
  3. 2020Learning Off-Policy with Online Planningcited by 53
  4. 2020Bandit optimisation of functions in the Mat'ern kernel RKHScited by 40
  5. 2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 33
  6. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  7. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  8. 2022End-to-End Learning to Warm-Start for Real-Time Quadratic Optimizationcited by 26
  9. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  10. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  11. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  12. 2025Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lenscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking
  2. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  3. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  4. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learning
  5. 2024Preference-Guided Reinforcement Learning for Efficient Exploration
  6. 2024Uncertainty-Aware Reward-Free Exploration with General Function Approximation
  7. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  8. 2025Uncertainty-aware Reward Design Process