# A Conceptual Framework for Externally-influenced Agents: An Assisted Reinforcement Learning Review

arXiv: https://arxiv.org/abs/2007.01544 · HTML: https://mlsanity.com/paper/2007.01544/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Persistent Rule-based Interactive Reinforcement Learning](https://mlsanity.com/paper/2102.02441/) (2021)
- [Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2505.19532/) (2025)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym](https://mlsanity.com/paper/2102.02639/) (2021)
- [Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning](https://mlsanity.com/paper/2312.15160/) (2023)
- [Broad-persistent Advice for Interactive Reinforcement Learning Scenarios](https://mlsanity.com/paper/2210.05187/) (2022)
- [Addressing the issue of stochastic environments and local decision-making in multi-objective reinforcement learning](https://mlsanity.com/paper/2211.08669/) (2022)
- [Reinforcement Learning for UAV control with Policy and Reward Shaping](https://mlsanity.com/paper/2212.03828/) (2022)
- [Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning](https://mlsanity.com/paper/2311.16807/) (2023)
- [Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations](https://mlsanity.com/paper/2312.09950/) (2023)
- [A Broad-persistent Advising Approach for Deep Interactive Reinforcement Learning in Robotic Environments](https://mlsanity.com/paper/2110.08003/) (2021)
- [Augmenting Flight Training with AI to Efficiently Train Pilots](https://mlsanity.com/paper/2210.06683/) (2022)
- [Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.11483/) (2023)
- [Human-Machine Teaming for UAVs: An Experimentation Platform](https://mlsanity.com/paper/2312.11718/) (2023)
- [Learning from Observation: A Survey of Recent Advances](https://mlsanity.com/paper/2509.19379/) (2025)

## Abstract (arXiv)

A long-term goal of reinforcement learning agents is to be able to perform tasks in complex real-world scenarios. The use of external information is one way of scaling agents to more complex problems. However, there is a general lack of collaboration or interoperability between different approaches using external information. In this work, while reviewing externally-influenced methods, we propose a conceptual framework and taxonomy for assisted reinforcement learning, aimed at fostering collaboration by classifying and comparing various methods that use external information in the learning process. The proposed taxonomy details the relationship between the external information source and the learner agent, highlighting the process of information decomposition, structure, retention, and how it can be used to influence agent learning. As well as reviewing state-of-the-art methods, we identify current streams of reinforcement learning that use external information in order to improve the agent's performance and its decision-making process. These include heuristic reinforcement learning, interactive reinforcement learning, learning from demonstration, transfer learning, and learning from multiple sources, among others. These streams of reinforcement learning operate with the shared objective of scaffolding the learner agent. Lastly, we discuss further possibilities for future work in the field of assisted reinforcement learning systems.