# Potential-Based Advice for Stochastic Policy Learning

arXiv: https://arxiv.org/abs/1907.08823 · HTML: https://mlsanity.com/paper/1907.08823/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2019.

## Built on by (most cited first, 3 of 3)

- [Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback](https://mlsanity.com/paper/2006.16498/) (2020)
- [Shaping Advice in Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2103.15941/) (2021)
- [Shaping Advice in Deep Reinforcement Learning](https://mlsanity.com/paper/2202.09489/) (2022)

## Abstract (arXiv)

This paper augments the reward received by a reinforcement learning agent with potential functions in order to help the agent learn (possibly stochastic) optimal policies. We show that a potential-based reward shaping scheme is able to preserve optimality of stochastic policies, and demonstrate that the ability of an agent to learn an optimal policy is not affected when this scheme is augmented to soft Q-learning. We propose a method to impart potential based advice schemes to policy gradient algorithms. An algorithm that considers an advantage actor-critic architecture augmented with this scheme is proposed, and we give guarantees on its convergence. Finally, we evaluate our approach on a puddle-jump grid world with indistinguishable states, and the continuous state and action mountain car environment from classical control. Our results indicate that these schemes allow the agent to learn a stochastic optimal policy faster and obtain a higher average reward.