# Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning

arXiv: https://arxiv.org/abs/2104.06411 · HTML: https://mlsanity.com/paper/2104.06411/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Playing hard exploration games by watching YouTube](https://mlsanity.com/paper/1805.11592/) (2018)
- [Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards](https://mlsanity.com/paper/1910.04450/) (2019)
- [Learning from Trajectories via Subgoal Discovery](https://mlsanity.com/paper/1911.07224/) (2019)
- [Improving interactive reinforcement learning: What makes a good teacher?](https://mlsanity.com/paper/1904.06879/) (2019)

## Built on by (most cited first, 11 of 11)

- [Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning](https://mlsanity.com/paper/2601.18984/) (2026)
- [Guaranteeing Control Requirements via Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2311.10026/) (2023)
- [Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach](https://mlsanity.com/paper/2302.02179/) (2023)
- [A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agents](https://mlsanity.com/paper/2305.16621/) (2023)
- [Learning user-defined sub-goals using memory editing in reinforcement learning](https://mlsanity.com/paper/2205.00399/) (2022)
- [Reinforcement Learning for UAV control with Policy and Reward Shaping](https://mlsanity.com/paper/2212.03828/) (2022)
- [A Fully Controllable Agent in the Path Planning using Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2205.09967/) (2022)
- [Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behavior](https://mlsanity.com/paper/2212.03733/) (2022)
- [STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order](https://mlsanity.com/paper/2601.08107/) (2026)
- [A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2608.08158/) (2026)
- [Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic](https://mlsanity.com/paper/2608.30406/) (2026)

## Abstract (arXiv)

Reinforcement learning, which acquires a policy maximizing long-term rewards, has been actively studied. Unfortunately, this learning type is too slow and difficult to use in practical situations because the state-action space becomes huge in real environments. Many studies have incorporated human knowledge into reinforcement Learning. Though human knowledge on trajectories is often used, a human could be asked to control an AI agent, which can be difficult. Knowledge on subgoals may lessen this requirement because humans need only to consider a few representative states on an optimal trajectory in their minds. The essential factor for learning efficiency is rewards. Potential-based reward shaping is a basic method for enriching rewards. However, it is often difficult to incorporate subgoals for accelerating learning over potential-based reward shaping. This is because the appropriate potentials are not intuitive for humans. We extend potential-based reward shaping and propose a subgoal-based reward shaping. The method makes it easier for human trainers to share their knowledge of subgoals. To evaluate our method, we obtained a subgoal series from participants and conducted experiments in three domains, four-rooms(discrete states and discrete actions), pinball(continuous and discrete), and picking(both continuous). We compared our method with a baseline reinforcement learning algorithm and other subgoal-based methods, including random subgoal and naive subgoal-based reward shaping. As a result, we found out that our reward shaping outperformed all other methods in learning efficiency.