mlsanityData to 14 Oct 2026

Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop

arXiv 2210.03455 ↗First posted 7 Oct 2022cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Human-in-the-loop (HiL) reinforcement learning is gaining traction in domains with large action and state spaces, and sparse rewards by allowing the agent to take advice from HiL. Beyond advice accommodation, a sequential decision-making agent must be able to express the extent to which it was able to utilize the human advice. Subsequently, the agent should provide a means for the HiL to inspect parts of advice that it had to reject in favor of the overall environment objective. We introduce the problem of Advice-Conformance Verification which requires reinforcement learning (RL) agents to provide assurances to the human in the loop regarding how much of their advice is being conformed to. We then propose a Tree-based lingua-franca to support this communication, called a Preference Tree. We study two cases of good and bad advice scenarios in MuJoCo's Humanoid environment. Through our experiments, we show that our method can provide an interpretable means of solving the Advice-Conformance Verification problem by conveying whether or not the agent is using the human's advice. Finally, we present a human-user study with 20 participants that validates our method.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 7 of its 15 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

7 of 15 references.

References · 7

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  3. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  4. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  5. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 41
  6. 2020FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedbackcited by 15
  7. 2021Action Advising with Advice Imitation in Deep Reinforcement Learningcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1.

5 papers.

  1. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learningcited by 4
  2. 2023Data Driven Reward Initialization for Preference based Reinforcement Learningcited by 3
  3. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  4. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferencescited by 2
  5. 2023Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environmentscited by 0