On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
The use of Potential-Based Reward Shaping (PBRS) has shown great promise in the ongoing research effort to tackle sample inefficiency in Reinforcement Learning (RL). However, choosing the right potential function remains an open challenge. Additionally, RL techniques are usually constrained to use a finite horizon for computational limitations, which introduces a bias when using PBRS. In this paper, we first build some theoretically-grounded intuition on why selecting the potential function as the optimal value function of the task at hand produces performance advantages. We then analyse the bias induced by finite horizons in the context of PBRS producing novel insights. Finally, leveraging abstractions as a way to approximate the optimal value function of the given task, we assess the sample efficiency and performance impact of PBRS on four environments including a goal-oriented navigation task and three Arcade Learning Environments (ALE) games. Remarkably, experimental results show that we can reach the same level of performance as CNN-based solutions with a simple fully-connected network.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 49 references that are in the map. First posted on arXiv 11 Apr 2024.
Builds on
18 of 49 references.
Closest ancestors · 2
Same direction, newest first
- 2024Potential-Based Reward Shaping For Intrinsic Motivation12 Feb
- 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning23 Jan
Other references · 16
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
- 2019Unsupervised State Representation Learning in Ataricited by 144
- 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
- 2018Stochastic Variance-Reduced Policy Gradientcited by 106
- 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
- 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
- 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
- 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 62
- 2017Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundarycited by 58
- 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
- 2021Heuristic-Guided Reinforcement Learningcited by 42
4 more
- 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 37
- 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
- 2022An Analysis of Model-Based Reinforcement Learning From Abstracted Observationscited by 2
- 2023Exploiting Multiple Abstractions in Episodic RL via Reward Shapingcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning
- 2024Potential-Based Intrinsic Motivation: Preserving Optimality With Complex, Non-Markovian Shaping Rewards
- 2026A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning
- 2026Automating Potential-based Reward Shaping with Vision Language Model Guidance
- 2025Bootstrapped Reward Shaping
- 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning
- 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
- 2024From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge