# Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation

arXiv: https://arxiv.org/abs/2006.14804 · HTML: https://mlsanity.com/paper/2006.14804/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2020.

## Built on by (most cited first, 30 of 42)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings](https://mlsanity.com/paper/2305.02317/) (2023)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems](https://mlsanity.com/paper/2109.09904/) (2021)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity](https://mlsanity.com/paper/2202.02886/) (2022)
- [Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?](https://mlsanity.com/paper/2401.05302/) (2024)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [Learning from Active Human Involvement through Proxy Value Propagation](https://mlsanity.com/paper/2502.03369/) (2025)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback](https://mlsanity.com/paper/2308.04332/) (2023)
- [Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning](https://mlsanity.com/paper/2310.18247/) (2023)
- [A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback](https://mlsanity.com/paper/2411.13410/) (2024)
- [Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions](https://mlsanity.com/paper/2307.10524/) (2023)
- [NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities](https://mlsanity.com/paper/2311.01454/) (2023)
- [Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences](https://mlsanity.com/paper/2210.15906/) (2022)
- [Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates](https://mlsanity.com/paper/2310.17786/) (2023)
- [RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation](https://mlsanity.com/paper/2405.03064/) (2024)
- [MAPLE: A Framework for Active Preference Learning Guided by Large Language Models](https://mlsanity.com/paper/2412.07207/) (2024)
- [Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents](https://mlsanity.com/paper/2601.18217/) (2026)
- [Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop](https://mlsanity.com/paper/2210.03455/) (2022)
- [Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AI](https://mlsanity.com/paper/2212.09667/) (2022)
- [Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation](https://mlsanity.com/paper/2211.10861/) (2022)
- [Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08738/) (2023)
- [Data-Efficient Learning from Human Interventions for Mobile Robots](https://mlsanity.com/paper/2503.04969/) (2025)
- [Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledge](https://mlsanity.com/paper/2202.07138/) (2022)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [Data Driven Reward Initialization for Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08733/) (2023)

## Abstract (arXiv)

Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.