mlsanityData to 14 Oct 2026

S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning

arXiv 2408.12130 ↗First posted 22 Aug 2024cs.AIcited by 4

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) stands out by utilizing human preferences as a direct reward signal, eliminating the need for intricate reward engineering. However, despite its potential, traditional PbRL methods are often constrained by the indistinguishability of segments, which impedes the learning process. In this paper, we introduce Skill-Enhanced Preference Optimization Algorithm (S-EPOA), which addresses the segment indistinguishability issue by integrating skill mechanisms into the preference learning framework. Specifically, we first conduct the unsupervised pretraining to learn useful skills. Then, we propose a novel query selection mechanism to balance the information gain and distinguishability over the learned skill space. Experimental results on a range of tasks, including robotic manipulation and locomotion, demonstrate that S-EPOA significantly outperforms conventional PbRL methods in terms of both robustness and learning efficiency. The results highlight the effectiveness of skill-driven learning in overcoming the challenges posed by segment indistinguishability.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 42 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

32 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences27 Feb

Other references · 31

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018DeepMind Control Suitecited by 956
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  10. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  11. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
19 more
  1. 2019Self-Supervised Exploration via Disagreementcited by 265
  2. 2017Count-Based Exploration with Neural Density Modelscited by 257
  3. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  4. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  5. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  6. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  7. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  8. 2021APS: Active Pretraining with Successor Featurescited by 112
  9. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  10. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  11. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  12. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  13. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  14. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  15. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  16. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  17. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  18. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  19. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  2. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  3. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  4. 2026Multi-Agent Reinforcement Learning via Agent-Specific Preferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
  2. 2026PAWS: Preference Learning with Advantage-Weighted Segments
  3. 2025DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition
  4. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
  5. 2024Advances in Preference-based Reinforcement Learning: A Review
  6. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning
  7. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences
  8. 2023Direct Preference-based Policy Optimization without Reward Modeling