mlsanityData to 14 Oct 2026

PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning

arXiv 2506.13741 ↗First posted 16 Jun 2025cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions. However, current PbRL methods face a critical challenge in effectively exploring the preference space, often converging prematurely to suboptimal policies that satisfy only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. We demonstrate that maintaining a diverse population of agents enables more comprehensive exploration of the preference landscape compared to single-agent approaches. Crucially, this diversity improves reward model learning by generating preference queries with clearly distinguishable behaviors, a key factor in real-world scenarios where humans must easily differentiate between options to provide meaningful feedback. Our experiments reveal that current methods may fail by getting stuck in local optima, requiring excessive feedback, or degrading significantly when human evaluators make errors on similar trajectories, a realistic scenario often overlooked by methods relying on perfect oracle teachers. Our population-based approach demonstrates robust performance when teachers mislabel similar trajectory segments and shows significantly enhanced preference exploration capabilities,particularly in environments with complex reward landscapes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 46 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

31 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations9 May
  2. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences27 Feb

Other references · 29

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2018DeepMind Control Suitecited by 956
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  5. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  6. 2015Illuminating search spaces by mapping elitescited by 425
  7. 2017Population Based Training of Neural Networkscited by 367
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  9. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  10. 2017Inverse Reward Designcited by 249
  11. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  12. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
17 more
  1. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  3. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  4. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  5. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  6. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  7. 2019Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptorscited by 57
  8. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  9. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  10. 2023Evolutionary Reinforcement Learning: A Surveycited by 29
  11. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  12. 2022Bayesian Generational Population-Based Trainingcited by 20
  13. 2021Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Informationcited by 19
  14. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
  15. 2023JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for tradingcited by 14
  16. 2021Faster Improvement Rate Population Based Trainingcited by 13
  17. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  2. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning
  3. 2024Advances in Preference-based Reinforcement Learning: A Review
  4. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
  5. 2024Preference-Guided Reinforcement Learning for Efficient Exploration
  6. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  7. 2025Best Policy Learning from Trajectory Preference Feedback
  8. 2023Direct Preference-based Policy Optimization without Reward Modeling