# Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes

arXiv: https://arxiv.org/abs/1805.04419 · HTML: https://mlsanity.com/paper/1805.04419/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2018.

## Built on by (most cited first, 8 of 8)

- [AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments](https://mlsanity.com/paper/2210.07940/) (2022)
- [CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments](https://mlsanity.com/paper/2306.04047/) (2023)
- [Periodic agent-state based Q-learning for POMDPs](https://mlsanity.com/paper/2407.06121/) (2024)
- [Design and implementation of an environment for Learning to Run a Power Network (L2RPN)](https://mlsanity.com/paper/2104.04080/) (2021)
- [Automatic Deduction Path Learning via Reinforcement Learning with Environmental Correction](https://mlsanity.com/paper/2306.10083/) (2023)
- [Uncertainty Maximization in Partially Observable Domains: A Cognitive Perspective](https://mlsanity.com/paper/2102.11232/) (2021)
- [Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2110.12810/) (2021)
- [MDP modeling for multi-stage stochastic programs](https://mlsanity.com/paper/2509.22981/) (2025)

## Abstract (arXiv)

In recent years, reinforcement learning has achieved many remarkable successes due to the growing adoption of deep learning techniques and the rapid growth in computing power. Nevertheless, it is well-known that flat reinforcement learning algorithms are often not able to learn well and data-efficient in tasks having hierarchical structures, e.g. consisting of multiple subtasks. Hierarchical reinforcement learning is a principled approach that is able to tackle these challenging tasks. On the other hand, many real-world tasks usually have only partial observability in which state measurements are often imperfect and partially observable. The problems of RL in such settings can be formulated as a partially observable Markov decision process (POMDP). In this paper, we study hierarchical RL in POMDP in which the tasks have only partial observability and possess hierarchical properties. We propose a hierarchical deep reinforcement learning approach for learning in hierarchical POMDP. The deep hierarchical RL algorithm is proposed to apply to both MDP and POMDP learning. We evaluate the proposed algorithm on various challenging hierarchical POMDP.