# Action Advising with Advice Imitation in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2104.08441 · HTML: https://mlsanity.com/paper/2104.08441/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 18 of 18)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Reinforcement Learning from Imperfect Demonstrations](https://mlsanity.com/paper/1802.05313/) (2018)
- [Learning to Teach in Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/1805.07830/) (2018)
- [Teaching on a Budget in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1905.01357/) (2019)
- [Learning to Teach Reinforcement Learning Agents](https://mlsanity.com/paper/1707.09079/) (2017)
- [Active Deep Q-learning with Demonstration](https://mlsanity.com/paper/1812.02632/) (2018)
- [Student-Initiated Action Advising via Advice Novelty](https://mlsanity.com/paper/2010.00381/) (2020)
- [Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfer](https://mlsanity.com/paper/1604.03986/) (2016)

## Built on by (most cited first, 10 of 10)

- [Explainable Action Advising for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.07882/) (2022)
- [Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop](https://mlsanity.com/paper/2210.03455/) (2022)
- [CANDERE-COACH: Reinforcement Learning from Noisy Feedback](https://mlsanity.com/paper/2409.15521/) (2024)
- [Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards](https://mlsanity.com/paper/2410.05782/) (2024)
- [Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning](https://mlsanity.com/paper/2312.15160/) (2023)
- [Methodical Advice Collection and Reuse in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07254/) (2022)
- [Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning](https://mlsanity.com/paper/2311.16807/) (2023)
- [Promoting Cooperation in Multi-Agent Reinforcement Learning via Mutual Help](https://mlsanity.com/paper/2302.09277/) (2023)
- [CueLearner: Bootstrapping and local policy adaptation from relative feedback](https://mlsanity.com/paper/2507.04730/) (2025)
- [Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning](https://mlsanity.com/paper/2508.09277/) (2025)

## Abstract (arXiv)

Action advising is a peer-to-peer knowledge exchange technique built on the teacher-student paradigm to alleviate the sample inefficiency problem in deep reinforcement learning. Recently proposed student-initiated approaches have obtained promising results. However, due to being in the early stages of development, these also have some substantial shortcomings. One of the abilities that are absent in the current methods is further utilising advice by reusing, which is especially crucial in the practical settings considering the budget and cost constraints in peer-to-peer. In this study, we present an approach to enable the student agent to imitate previously acquired advice to reuse them directly in its exploration policy, without any interventions in the learning mechanism itself. In particular, we employ a behavioural cloning module to imitate the teacher policy and use dropout regularisation to have a notion of epistemic uncertainty to keep track of which state-advice pairs are actually collected. As the results of experiments we conducted in three Atari games show, advice reusing via generalisation is indeed a feasible option in deep RL and our approach can successfully achieve this while significantly improving the learning performance, even when paired with a simple early advising heuristic.