mlsanityData to 7 Oct 2026

Bootstrapped Reward Shaping

arXiv 2501.00989 ↗First posted 2 Jan 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required "potential function" must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a "bootstrapped" method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

16 of 33 references.

References · 16

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
  5. 2020Quantifying Differences in Reward Functionscited by 62
  6. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  7. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  8. 2021Identifiability in inverse reinforcement learningcited by 55
  9. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
  10. 2020Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Taskscited by 28
  11. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  12. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
4 more
  1. 2022Calculus on MDPs: Potential Shaping as a Gradientcited by 4
  2. 2024Boosting Soft Q-Learning by Boundingcited by 4
  3. 2022Soft Action Priors: Towards Robust Policy Transfercited by 2
  4. 2023Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shapingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0
  2. 2026SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learningcited by 0
  3. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0