mlsanityData to 7 Oct 2026

Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion

arXiv 2210.15096 ↗First posted 27 Oct 2022cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

There is a growing interest in developing automated agents that can work alongside humans. In addition to completing the assigned task, such an agent will undoubtedly be expected to behave in a manner that is preferred by the human. This requires the human to communicate their preferences to the agent. To achieve this, the current approaches either require the users to specify the reward function or the preference is interactively learned from queries that ask the user to compare behavior. The former approach can be challenging if the internal representation used by the agent is inscrutable to the human while the latter is unnecessarily cumbersome for the user if their preference can be specified more easily in symbolic terms. In this work, we propose PRESCA (PREference Specification through Concept Acquisition), a system that allows users to specify their preferences in terms of concepts that they understand. PRESCA maintains a set of such concepts in a shared vocabulary. If the relevant concept is not in the shared vocabulary, then it is learned. To make learning a new concept more feedback efficient, PRESCA leverages causal associations between the target concept and concepts that are already known. In addition, we use a novel data augmentation approach to further reduce required feedback. We evaluate PRESCA by using it on a Minecraft environment and show that it can effectively align the agent with the user's preference.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 17 of its 42 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

17 of 42 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  6. 2017Inverse Reward Designcited by 246
  7. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  8. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  9. 2018The Contextual Loss for Image Transformation with Non-Aligned Datacited by 135
  10. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  11. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 77
  12. 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 51
5 more
  1. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
  2. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  3. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  4. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  5. 2020Active Learning for Skewed Data Setscited by 6

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2024MAPLE: A Framework for Active Preference Learning Guided by Large Language Modelscited by 6
  2. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learningcited by 4
  3. 2023Data Driven Reward Initialization for Preference based Reinforcement Learningcited by 3
  4. 2023Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environmentscited by 0