# Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks

arXiv: https://arxiv.org/abs/2107.06661 · HTML: https://mlsanity.com/paper/2107.06661/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 34 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 5 of 5)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Learning Montezuma's Revenge from a Single Demonstration](https://mlsanity.com/paper/1812.03381/) (2018)

## Built on by (most cited first, 6 of 6)

- [Calculus on MDPs: Potential Shaping as a Gradient](https://mlsanity.com/paper/2208.09570/) (2022)
- [Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework](https://mlsanity.com/paper/2409.04744/) (2024)
- [Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics](https://mlsanity.com/paper/2111.07908/) (2021)
- [D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioning](https://mlsanity.com/paper/2210.14428/) (2022)
- [Exploiting Multiple Abstractions in Episodic RL via Reward Shaping](https://mlsanity.com/paper/2303.00516/) (2023)
- [Path-conditioned Reinforcement Learning-based Local Planning for Long-Range Navigation](https://mlsanity.com/paper/2603.13888/) (2026)

## Abstract (arXiv)

In high-dimensional state spaces, the usefulness of Reinforcement Learning (RL) is limited by the problem of exploration. This issue has been addressed using potential-based reward shaping (PB-RS) previously. In the present work, we introduce Final-Volume-Preserving Reward Shaping (FV-RS). FV-RS relaxes the strict optimality guarantees of PB-RS to a guarantee of preserved long-term behavior. Being less restrictive, FV-RS allows for reward shaping functions that are even better suited for improving the sample efficiency of RL algorithms. In particular, we consider settings in which the agent has access to an approximate plan. Here, we use examples of simulated robotic manipulation tasks to demonstrate that plan-based FV-RS can indeed significantly improve the sample efficiency of RL over plan-based PB-RS.