Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
Not on the map. Why?
Abstract (arXiv)
Preference-Based reinforcement learning (PBRL) learns directly from the preferences of human teachers regarding agent behaviors without needing meticulously designed reward functions. However, existing PBRL methods often learn primarily from explicit preferences, neglecting the possibility that teachers may choose equal preferences. This neglect may hinder the understanding of the agent regarding the task perspective of the teacher, leading to the loss of important information. To address this issue, we introduce the Equal Preference Learning Task, which optimizes the neural network by promoting similar reward predictions when the behaviors of two agents are labeled as equal preferences. Building on this task, we propose a novel PBRL method, Multi-Type Preference Learning (MTPL), which allows simultaneous learning from equal preferences while leveraging existing methods for learning from explicit preferences. To validate our approach, we design experiments applying MTPL to four existing state-of-the-art baselines across ten locomotion and robotic manipulation tasks in the DeepMind Control Suite. The experimental results indicate that simultaneous learning from both equal and explicit preferences enables the PBRL method to more comprehensively understand the feedback from teachers, thereby enhancing feedback efficiency. Project page: urlhttps://github.com/FeiCuiLengMMbb/paperMTPL
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 45 references that are in the map. First posted on arXiv 11 Sep 2024.
Builds on
23 of 45 references.
References · 23
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2018DeepMind Control Suitecited by 935
- 2017A Survey on Multi-Task Learningcited by 838
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
- 2017Inverse Reward Designcited by 246
- 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
- 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
- 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
- 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 91
11 more
- 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
- 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
- 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 74
- 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
- 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
- 2018Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methodscited by 32
- 2024Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Erascited by 26
- 2020Avoiding Side Effects in Complex Environmentscited by 24
- 2023Rating-based Reinforcement Learningcited by 14
- 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
- 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learningcited by 4
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.