# Creativity of AI: Hierarchical Planning Model Learning for Facilitating Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2112.09836 · HTML: https://mlsanity.com/paper/2112.09836/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 18 Dec 2021.

## Builds on (most cited first, 10 of 10)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles](https://mlsanity.com/paper/2001.11231/) (2020)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making](https://mlsanity.com/paper/1804.07779/) (2018)
- [SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning](https://mlsanity.com/paper/2012.13037/) (2020)
- [Learning Portable Representations for High-Level Planning](https://mlsanity.com/paper/1905.12006/) (2019)
- [Transfer Value Iteration Networks](https://mlsanity.com/paper/1911.05701/) (2019)
- [Refining HTN Methods via Task Insertion with Preferences](https://mlsanity.com/paper/1911.12949/) (2019)

## Built on by (most cited first, 1 of 1)

- [Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learning](https://mlsanity.com/paper/2503.21406/) (2025)

## Abstract (arXiv)

Despite of achieving great success in real-world applications, Deep Reinforcement Learning (DRL) is still suffering from three critical issues, i.e., data efficiency, lack of the interpretability and transferability. Recent research shows that embedding symbolic knowledge into DRL is promising in addressing those challenges. Inspired by this, we introduce a novel deep reinforcement learning framework with symbolic options. Our framework features a loop training procedure, which enables guiding the improvement of policy by planning with planning models (including action models and hierarchical task network models) and symbolic options learned from interactive trajectories automatically. The learned symbolic options alleviate the dense requirement of expert domain knowledge and provide inherent interpretability of policies. Moreover, the transferability and data efficiency can be further improved by planning with the symbolic planning models. To validate the effectiveness of our framework, we conduct experiments on two domains, Montezuma's Revenge and Office World, respectively. The results demonstrate the comparable performance, improved data efficiency, interpretability and transferability.