mlsanityData to 14 Oct 2026

Two-Step Offline Preference-Based Reinforcement Learning with Constrained Actions

arXiv 2401.00330 ↗First posted 30 Dec 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference-based reinforcement learning (PBRL) in the offline setting has succeeded greatly in industrial applications such as chatbots. A two-step learning framework where one applies a reinforcement learning step after a reward modeling step has been widely adopted for the problem. However, such a method faces challenges from the risk of reward hacking and the complexity of reinforcement learning. To overcome the challenge, our insight is that both challenges come from the state-actions not supported in the dataset. Such state-actions are unreliable and increase the complexity of the reinforcement learning problem at the second step. Based on the insight, we develop a novel two-step learning method called PRC: preference-based reinforcement learning with constrained actions. The high-level idea is to limit the reinforcement learning agent to optimize over a constrained action space that excludes the out-of-distribution state-actions. We empirically verify that our method has high learning efficiency on various datasets in robotic control environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

19 of 31 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  9. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  10. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  11. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  12. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
7 more
  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  2. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  3. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  5. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 98
  6. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Direct Preference-based Policy Optimization without Reward Modeling
  2. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences
  3. 2024Advances in Preference-based Reinforcement Learning: A Review
  4. 2025Offline Safe Policy Optimization From Heterogeneous Feedback
  5. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  6. 2023Beyond Reward: Offline Preference-guided Policy Optimization
  7. 2024Online Policy Learning from Offline Preferences
  8. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning