mlsanityData to 14 Oct 2026

SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning

arXiv 2506.14648 ↗First posted 17 Jun 2025cs.RO cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Preference-based Reinforcement Learning (PbRL) methods provide a solution to avoid reward engineering by learning reward models based on human preferences. However, poor feedback- and sample- efficiency still remain the problems that hinder the application of PbRL. In this paper, we present a novel efficient query selection and preference-guided exploration method, called SENIOR, which could select the meaningful and easy-to-comparison behavior segment pairs to improve human feedback-efficiency and accelerate policy learning with the designed preference-guided intrinsic rewards. Our key idea is twofold: (1) We designed a Motion-Distinction-based Selection scheme (MDS). It selects segment pairs with apparent motion and different directions through kernel density estimation of states, which is more task-related and easy for human preference labeling; (2) We proposed a novel preference-guided exploration method (PGE). It encourages the exploration towards the states with high preference and low visits and continuously guides the agent achieving the valuable samples. The synergy between the two mechanisms could significantly accelerate the progress of reward and policy learning. Our experiments show that SENIOR outperforms other five existing methods in both human feedback-efficiency and policy convergence speed on six complex robot manipulation tasks from simulation and four real-worlds. Videos can be found on our project website: https://2025senior.github.io/

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 41 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

29 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning17 Feb
  2. 2023Data Driven Reward Initialization for Preference based Reinforcement Learning17 Feb
  3. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences17 Oct
  4. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2018Exploration by Random Network Distillationcited by 993
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  9. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  10. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  11. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  12. 2019Self-Supervised Exploration via Disagreementcited by 265
10 more
  1. 2017Count-Based Exploration with Neural Density Modelscited by 257
  2. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  3. 2019Efficient Exploration via State Marginal Matchingcited by 188
  4. 2017Parameter Space Noise for Explorationcited by 161
  5. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  6. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 57
  7. 2019Curiosity-Driven Experience Prioritization via Density Estimationcited by 27
  8. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  9. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
  10. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Multi-Agent Reinforcement Learning via Agent-Specific Preferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  2. 2024Preference-Guided Reinforcement Learning for Efficient Exploration
  3. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  4. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  5. 2024Advances in Preference-based Reinforcement Learning: A Review
  6. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  7. 2023Direct Preference-based Policy Optimization without Reward Modeling
  8. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning