mlsanityData to 7 Oct 2026

Preference-Guided Reinforcement Learning for Efficient Exploration

arXiv 2407.06503 ↗First posted 9 Jul 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate preference-based reinforcement learning (PbRL), which enables reinforcement learning (RL) agents to learn from human feedback. This is particularly valuable when defining a fine-grain reward function is not feasible. However, this approach is inefficient and impractical for promoting deep exploration in hard-exploration tasks with long horizons and sparse rewards. To tackle this issue, we introduce LOPE: Learning Online with trajectory Preference guidancE, an end-to-end preference-guided RL framework that enhances exploration efficiency in hard-exploration tasks. Our intuition is that LOPE directly adjusts the focus of online exploration by considering human feedback as guidance, thereby avoiding the need to learn a separate reward model from preferences. Specifically, LOPE includes a two-step sequential policy optimization technique consisting of trust-region-based policy improvement and preference guidance steps. We reformulate preference guidance as a trajectory-wise state marginal matching problem that minimizes the maximum mean discrepancy distance between the preferred trajectories and the learned policy. Furthermore, we provide a theoretical analysis to characterize the performance improvement bound and evaluate the effectiveness of the LOPE. When assessed in various challenging hard-exploration environments, LOPE outperforms several state-of-the-art methods in terms of convergence rate and overall performance.The code used in this study is available at https://github.com/buaawgj/LOPE.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 68 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

47 of 68 references.

References · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2016OpenAI Gymcited by 1,684
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
35 more
  1. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  2. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  4. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  5. 2017Noisy Networks for Explorationcited by 262
  6. 2017Inverse Reward Designcited by 246
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  8. 2019Efficient Exploration via State Marginal Matchingcited by 186
  9. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  10. 2018Self-Imitation Learningcited by 135
  11. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  12. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  13. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  14. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  15. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  16. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  17. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  18. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  19. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  20. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  21. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  22. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  23. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  24. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  25. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  26. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  27. 2018Learning Self-Imitating Diverse Policiescited by 29
  28. 2020Self-Concordant Analysis of Generalized Linear Bandits with Forgettingcited by 29
  29. 2018Generative Adversarial Self-Imitation Learningcited by 25
  30. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  31. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
  32. 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstrationcited by 19
  33. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  34. 2024Learning Diverse Policies with Soft Self-Generated Guidancecited by 3
  35. 2024Trajectory-Oriented Policy Optimization with Sparse Rewardscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning
  2. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  3. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
  4. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  5. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  6. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  7. 2025Best Policy Learning from Trajectory Preference Feedback
  8. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning