# Introspective Action Advising for Interpretable Transfer Learning

arXiv: https://arxiv.org/abs/2306.12314 · HTML: https://mlsanity.com/paper/2306.12314/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 72 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 30 of 39)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Online Meta-Learning](https://mlsanity.com/paper/1902.08438/) (2019)
- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Distilling Policy Distillation](https://mlsanity.com/paper/1902.02186/) (2019)
- [Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning](https://mlsanity.com/paper/2201.03529/) (2022)
- [Kickstarting Deep Reinforcement Learning](https://mlsanity.com/paper/1803.03835/) (2018)
- [Adversarially robust transfer learning](https://mlsanity.com/paper/1905.08232/) (2019)
- [Learning to Teach in Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/1805.07830/) (2018)

## Built on by (most cited first, 2 of 2)

- [Explaining Agent Behavior with Large Language Models](https://mlsanity.com/paper/2309.10346/) (2023)
- [Energy-Based Transfer for Reinforcement Learning](https://mlsanity.com/paper/2506.16590/) (2025)

## Abstract (arXiv)

Transfer learning can be applied in deep reinforcement learning to accelerate the training of a policy in a target task by transferring knowledge from a policy learned in a related source task. This is commonly achieved by copying pretrained weights from the source policy to the target policy prior to training, under the constraint that they use the same model architecture. However, not only does this require a robust representation learned over a wide distribution of states -- often failing to transfer between specialist models trained over single tasks -- but it is largely uninterpretable and provides little indication of what knowledge is transferred. In this work, we propose an alternative approach to transfer learning between tasks based on action advising, in which a teacher trained in a source task actively guides a student's exploration in a target task. Through introspection, the teacher is capable of identifying when advice is beneficial to the student and should be given, and when it is not. Our approach allows knowledge transfer between policies agnostic of the underlying representations, and we empirically show that this leads to improved convergence rates in Gridworld and Atari environments while providing insight into what knowledge is transferred.