# A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges

arXiv: https://arxiv.org/abs/2105.12949 · HTML: https://mlsanity.com/paper/2105.12949/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2021.

## Built on by (most cited first, 30 of 41)

- [Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment](https://mlsanity.com/paper/2211.08416/) (2022)
- [TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction](https://mlsanity.com/paper/2405.10315/) (2024)
- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning](https://mlsanity.com/paper/2210.10765/) (2022)
- [A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors](https://mlsanity.com/paper/2302.13137/) (2023)
- [RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback](https://mlsanity.com/paper/2308.04332/) (2023)
- [Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2210.11846/) (2022)
- [Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions](https://mlsanity.com/paper/2112.11230/) (2021)
- [Reward Learning from Multiple Feedback Types](https://mlsanity.com/paper/2502.21038/) (2025)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning](https://mlsanity.com/paper/2311.10284/) (2023)
- [Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities](https://mlsanity.com/paper/2408.12548/) (2024)
- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [Targeted Data Acquisition for Evolving Negotiation Agents](https://mlsanity.com/paper/2106.07728/) (2021)
- [Good Time to Ask: A Learning Framework for Asking for Help in Embodied Visual Navigation](https://mlsanity.com/paper/2206.10606/) (2022)
- [Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes](https://mlsanity.com/paper/2407.00364/) (2024)
- [From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP](https://mlsanity.com/paper/2510.12817/) (2025)
- [Towards Interactive Reinforcement Learning with Intrinsic Feedback](https://mlsanity.com/paper/2112.01575/) (2021)
- [Teachable Reinforcement Learning via Advice Distillation](https://mlsanity.com/paper/2203.11197/) (2022)
- [How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robots](https://mlsanity.com/paper/2407.06459/) (2024)
- [Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2504.05393/) (2025)
- [Grower-in-the-Loop Interactive Reinforcement Learning for Greenhouse Climate Control](https://mlsanity.com/paper/2505.23355/) (2025)
- [Quantifying the Effect of Feedback Frequency in Interactive Reinforcement Learning for Robotic Tasks](https://mlsanity.com/paper/2207.09845/) (2022)
- [Broad-persistent Advice for Interactive Reinforcement Learning Scenarios](https://mlsanity.com/paper/2210.05187/) (2022)
- [Establishing Shared Query Understanding in an Open Multi-Agent System](https://mlsanity.com/paper/2305.09349/) (2023)
- [Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systems](https://mlsanity.com/paper/2307.03853/) (2023)
- [Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Tasks](https://mlsanity.com/paper/2312.05991/) (2023)
- [MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention](https://mlsanity.com/paper/2406.16258/) (2024)
- [Incorporating Rivalry in Reinforcement Learning for a Competitive Game](https://mlsanity.com/paper/2208.10327/) (2022)
- [ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2301.09941/) (2023)

## Abstract (arXiv)

Interactive reinforcement learning (RL) has been successfully used in various applications in different fields, which has also motivated HCI researchers to contribute in this area. In this paper, we survey interactive RL to empower human-computer interaction (HCI) researchers with the technical background in RL needed to design new interaction techniques and propose new applications. We elucidate the roles played by HCI researchers in interactive RL, identifying ideas and promising research directions. Furthermore, we propose generic design principles that will provide researchers with a guide to effectively implement interactive RL applications.