# Methodical Advice Collection and Reuse in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.07254 · HTML: https://mlsanity.com/paper/2204.07254/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 30 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 6 of 6)

- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Generalization and Regularization in DQN](https://mlsanity.com/paper/1810.00123/) (2018)
- [Learning to Teach in Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/1805.07830/) (2018)
- [Action Advising with Advice Imitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2104.08441/) (2021)
- [Student-Initiated Action Advising via Advice Novelty](https://mlsanity.com/paper/2010.00381/) (2020)
- [Learning on a Budget via Teacher Imitation](https://mlsanity.com/paper/2104.08440/) (2021)

## Built on by (most cited first, 1 of 1)

- [Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning](https://mlsanity.com/paper/2311.16807/) (2023)

## Abstract (arXiv)

Reinforcement learning (RL) has shown great success in solving many challenging tasks via use of deep neural networks. Although using deep learning for RL brings immense representational power, it also causes a well-known sample-inefficiency problem. This means that the algorithms are data-hungry and require millions of training samples to converge to an adequate policy. One way to combat this issue is to use action advising in a teacher-student framework, where a knowledgeable teacher provides action advice to help the student. This work considers how to better leverage uncertainties about when a student should ask for advice and if the student can model the teacher to ask for less advice. The student could decide to ask for advice when it is uncertain or when both it and its model of the teacher are uncertain. In addition to this investigation, this paper introduces a new method to compute uncertainty for a deep RL agent using a secondary neural network. Our empirical results show that using dual uncertainties to drive advice collection and reuse may improve learning performance across several Atari games.