# Student-Initiated Action Advising via Advice Novelty

arXiv: https://arxiv.org/abs/2010.00381 · HTML: https://mlsanity.com/paper/2010.00381/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [Action Advising with Advice Imitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2104.08441/) (2021)
- [Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards](https://mlsanity.com/paper/2410.05782/) (2024)
- [Learning on a Budget via Teacher Imitation](https://mlsanity.com/paper/2104.08440/) (2021)
- [Methodical Advice Collection and Reuse in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07254/) (2022)
- [Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning](https://mlsanity.com/paper/2311.16807/) (2023)
- [Boosting deep Reinforcement Learning using pretraining with Logical Options](https://mlsanity.com/paper/2603.06565/) (2026)
- [Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2605.30576/) (2026)

## Abstract (arXiv)

Action advising is a budget-constrained knowledge exchange mechanism between teacher-student peers that can help tackle exploration and sample inefficiency problems in deep reinforcement learning (RL). Most recently, student-initiated techniques that utilise state novelty and uncertainty estimations have obtained promising results. However, the approaches built on these estimations have some potential weaknesses. First, they assume that the convergence of the student's RL model implies less need for advice. This can be misleading in scenarios with teacher absence early on where the student is likely to learn suboptimally by itself; yet also ignore the teacher's assistance later. Secondly, the delays between encountering states and having them to take effect in the RL model updates in presence of the experience replay dynamics cause a feedback lag in what the student actually needs advice for. We propose a student-initiated algorithm that alleviates these by employing Random Network Distillation (RND) to measure the novelty of a piece of advice. Furthermore, we perform RND updates only for the advised states to ensure that the student's own learning does not impair its ability to leverage the teacher. Experiments in GridWorld and MinAtar show that our approach performs on par with the state-of-the-art and demonstrates significant advantages in the scenarios where the existing methods are prone to fail.