mlsanityData to 14 Oct 2026

Online Policy Learning from Offline Preferences

arXiv 2403.10160 ↗First posted 15 Mar 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In preference-based reinforcement learning (PbRL), a reward function is learned from a type of human feedback called preference. To expedite preference collection, recent works have leveraged offline preferences, which are preferences collected for some offline data. In this scenario, the learned reward function is fitted on the offline data. If a learning agent exhibits behaviors that do not overlap with the offline data, the learned reward function may encounter generalizability issues. To address this problem, the present study introduces a framework that consolidates offline preferences and virtual preferences for PbRL, which are comparisons between the agent's behaviors and the offline data. Critically, the reward function can track the agent's behaviors using the virtual preferences, thereby offering well-aligned guidance to the agent. Through experiments on continuous control tasks, this study demonstrates the effectiveness of incorporating the virtual preferences in PbRL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

15 of 25 references.

References · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  6. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  7. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  8. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  9. 2021What Matters for Adversarial Imitation Learning?cited by 70
  10. 2018Shared Autonomy via Deep Reinforcement Learningcited by 66
  11. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  12. 2017Pragmatic-Pedagogic Value Alignmentcited by 36
3 more
  1. 2020Active Preference Learning using Maximum Regretcited by 21
  2. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  3. 2021Batch Reinforcement Learning from Crowdscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  2. 2023Direct Preference-based Policy Optimization without Reward Modeling
  3. 2023Beyond Reward: Offline Preference-guided Policy Optimization
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  5. 2024Preference Elicitation for Offline Reinforcement Learning
  6. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency
  7. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  8. 2025Offline Safe Policy Optimization From Heterogeneous Feedback