# Potential-based Reward Shaping in Sokoban

arXiv: https://arxiv.org/abs/2109.05022 · HTML: https://mlsanity.com/paper/2109.05022/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 22 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 6 of 6)

- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [An investigation of model-free planning](https://mlsanity.com/paper/1901.03559/) (2019)
- [On the role of planning in model-based deep reinforcement learning](https://mlsanity.com/paper/2011.04021/) (2020)
- [Learning to Search with MCTSnets](https://mlsanity.com/paper/1802.04697/) (2018)
- [Transfer Learning and Curriculum Learning in Sokoban](https://mlsanity.com/paper/2105.11702/) (2021)

## Built on by (most cited first, 1 of 1)

- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)

## Abstract (arXiv)

Learning to solve sparse-reward reinforcement learning problems is difficult, due to the lack of guidance towards the goal. But in some problems, prior knowledge can be used to augment the learning process. Reward shaping is a way to incorporate prior knowledge into the original reward function in order to speed up the learning. While previous work has investigated the use of expert knowledge to generate potential functions, in this work, we study whether we can use a search algorithm(A*) to automatically generate a potential function for reward shaping in Sokoban, a well-known planning task. The results showed that learning with shaped reward function is faster than learning from scratch. Our results indicate that distance functions could be a suitable function for Sokoban. This work demonstrates the possibility of solving multiple instances with the help of reward shaping. The result can be compressed into a single policy, which can be seen as the first phrase towards training a general policy that is able to solve unseen instances.