Action Advising with Advice Imitation in Deep Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Action advising is a peer-to-peer knowledge exchange technique built on the teacher-student paradigm to alleviate the sample inefficiency problem in deep reinforcement learning. Recently proposed student-initiated approaches have obtained promising results. However, due to being in the early stages of development, these also have some substantial shortcomings. One of the abilities that are absent in the current methods is further utilising advice by reusing, which is especially crucial in the practical settings considering the budget and cost constraints in peer-to-peer. In this study, we present an approach to enable the student agent to imitate previously acquired advice to reuse them directly in its exploration policy, without any interventions in the learning mechanism itself. In particular, we employ a behavioural cloning module to imitate the teacher policy and use dropout regularisation to have a notion of epistemic uncertainty to keep track of which state-advice pairs are actually collected. As the results of experiments we conducted in three Atari games show, advice reusing via generalisation is indeed a feasible option in deep RL and our approach can successfully achieve this while significantly improving the learning performance, even when paired with a simple early advising heuristic.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 17 Apr 2021.
Builds on
18 of 33 references.
References · 18
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2015Prioritized Experience Replaycited by 1,070
- 2018Exploration by Random Network Distillationcited by 966
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
- 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
- 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
- 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
6 more
- 2018Learning to Teach in Cooperative Multiagent Reinforcement Learningcited by 45
- 2019Teaching on a Budget in Multi-Agent Deep Reinforcement Learningcited by 14
- 2017Learning to Teach Reinforcement Learning Agentscited by 10
- 2018Active Deep Q-learning with Demonstrationcited by 10
- 2020Student-Initiated Action Advising via Advice Noveltycited by 7
- 2016Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfercited by 5
Built on by
Citing papers per year: 2022: 3; 2023: 3; 2024: 2; 2025: 2.
10 papers.
- 2022Explainable Action Advising for Multi-Agent Reinforcement Learningcited by 10
- 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loopcited by 5
- 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedbackcited by 3
- 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
- 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 2
- 2022Methodical Advice Collection and Reuse in Deep Reinforcement Learningcited by 1
- 2023Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learningcited by 1
- 2023Promoting Cooperation in Multi-Agent Reinforcement Learning via Mutual Helpcited by 0
- 2025CueLearner: Bootstrapping and local policy adaptation from relative feedbackcited by 0
- 2025Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learningcited by 0
- 2025Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning12 Aug 2025
- 2025CueLearner: Bootstrapping and local policy adaptation from relative feedback7 Jul 2025
- 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards8 Oct 2024
- 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedback23 Sep 2024
- 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning23 Dec 2023
- 2023Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning28 Nov 2023
- 2023Promoting Cooperation in Multi-Agent Reinforcement Learning via Mutual Help18 Feb 2023
- 2022Explainable Action Advising for Multi-Agent Reinforcement Learning15 Nov 2022
- 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop7 Oct 2022
- 2022Methodical Advice Collection and Reuse in Deep Reinforcement Learning14 Apr 2022
Similar, not cited
Close in wording, no citation either way
- 2022Teachable Reinforcement Learning via Advice Distillation
- 2022Data augmentation for efficient learning from parametric experts
- 2025Refined Policy Distillation: From VLA Generalists to RL Experts
- 2026When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
- 2020Bridging the Imitation Gap by Adaptive Insubordination
- 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
- 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning
- 2018Hierarchical Imitation and Reinforcement Learning