# Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning

arXiv: https://arxiv.org/abs/2502.01307 · HTML: https://mlsanity.com/paper/2502.01307/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 3 of its 18 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 3 of 3)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models](https://mlsanity.com/paper/2011.01298/) (2020)
- [Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping](https://mlsanity.com/paper/2209.07288/) (2022)

## Built on by (most cited first, 8 of 8)

- [Diversity-Incentivized Exploration for Versatile Reasoning](https://mlsanity.com/paper/2509.26209/) (2025)
- [Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning](https://mlsanity.com/paper/2509.26578/) (2025)
- [Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation](https://mlsanity.com/paper/2601.12401/) (2026)
- [Automating Potential-based Reward Shaping with Vision Language Model Guidance](https://mlsanity.com/paper/2606.27180/) (2026)
- [Quantum-Inspired Multi Agent Reinforcement Learning for Exploration Exploitation Optimization in UAV-Assisted 6G Network Deployment](https://mlsanity.com/paper/2512.20624/) (2025)
- [Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning](https://mlsanity.com/paper/2603.09331/) (2026)
- [A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2608.08158/) (2026)
- [Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic](https://mlsanity.com/paper/2608.30406/) (2026)

## Abstract (arXiv)

Potential-based reward shaping is commonly used to incorporate prior knowledge of how to solve the task into reinforcement learning because it can formally guarantee policy invariance. As such, the optimal policy and the ordering of policies by their returns are not altered by potential-based reward shaping. In this work, we highlight the dependence of effective potential-based reward shaping on the initial Q-values and external rewards, which determine the agent's ability to exploit the shaping rewards to guide its exploration and achieve increased sample efficiency. We formally derive how a simple linear shift of the potential function can be used to improve the effectiveness of reward shaping without changing the encoded preferences in the potential function, and without having to adjust the initial Q-values, which can be challenging and undesirable in deep reinforcement learning. We show the theoretical limitations of continuous potential functions for correctly assigning positive and negative reward shaping values. We verify our theoretical findings empirically on Gridworld domains with sparse and uninformative reward functions, as well as on the Cart Pole and Mountain Car environments, where we demonstrate the application of our results in deep reinforcement learning.