Potential-Based Intrinsic Motivation: Preserving Optimality With Complex, Non-Markovian Shaping Rewards
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Recently there has been a proliferation of intrinsic motivation (IM) reward-shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we prove preserves the set of optimal policies under a more general set of functions than has been previously proven. We also present em Potential-Based Intrinsic Motivation (PBIM) and em Generalized Reward Matching (GRM), methods for converting IM rewards into a potential-based form that are useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey and Cliff Walking environments, we demonstrate that PBIM and GRM successfully prevent the agent from converging to a suboptimal policy and can speed up training. Additionally, we prove that GRM is sufficiently general as to encompass all potential-based reward shaping functions. This paper expands on previous work introducing the PBIM method, and provides an extension to the more general method of GRM, as well as additional proofs, experimental results, and discussion.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 38 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
17 of 38 references.
Closest ancestors · 1
Same direction, newest first
Other references · 16
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2016Concrete Problems in AI Safetycited by 1,642
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
- 2017Hindsight Experience Replaycited by 1,136
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
- 2018Exploration by Random Network Distillationcited by 993
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
- 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
- 2017Count-Based Exploration with Neural Density Modelscited by 257
- 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
4 more
- 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
- 2019Optimal Policies Tend to Seek Powercited by 119
- 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
- 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping
- 2025Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning
- 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
- 2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning
- 2023Behavior Alignment via Reward Function Optimization
- 2023Generative Intrinsic Optimization: Intrinsic Control with Model Learning
- 2026Automating Potential-based Reward Shaping with Vision Language Model Guidance
- 2025Bootstrapped Reward Shaping