# Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks

arXiv: https://arxiv.org/abs/1901.10113 · HTML: https://mlsanity.com/paper/1901.10113/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 4 of 4)

- [A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots](https://mlsanity.com/paper/2103.08183/) (2021)
- [Goal-Directed Planning by Reinforcement Learning and Active Inference](https://mlsanity.com/paper/2106.09938/) (2021)
- [Compositional Learning of Visually-Grounded Concepts Using Reinforcement](https://mlsanity.com/paper/2309.04504/) (2023)
- [Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2112.05343/) (2021)

## Abstract (arXiv)

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.