# Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control

arXiv: https://arxiv.org/abs/2408.10970 · HTML: https://mlsanity.com/paper/2408.10970/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 7 of 7)

- [Neurosymbolic AI: The 3rd Wave](https://mlsanity.com/paper/2012.05876/) (2020)
- [On the Relationship Between Active Inference and Control as Inference](https://mlsanity.com/paper/2006.12964/) (2020)
- [pymdp: A Python library for active inference in discrete state spaces](https://mlsanity.com/paper/2201.03904/) (2022)
- [Recurrent switching linear dynamical systems](https://mlsanity.com/paper/1610.08466/) (2016)
- [DQN with model-based exploration: efficient learning on environments with sparse rewards](https://mlsanity.com/paper/1903.09295/) (2019)
- [Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation](https://mlsanity.com/paper/2005.01432/) (2020)
- [Model-Based Reinforcement Learning via Stochastic Hybrid Models](https://mlsanity.com/paper/2111.06211/) (2021)

## Abstract (arXiv)

An open problem in artificial intelligence is how systems can flexibly learn discrete abstractions that are useful for solving inherently continuous problems. Previous work has demonstrated that a class of hybrid state-space model known as recurrent switching linear dynamical systems (rSLDS) discover meaningful behavioural units via the piecewise linear decomposition of complex continuous dynamics (Linderman et al., 2016). Furthermore, they model how the underlying continuous states drive these discrete mode switches. We propose that the rich representations formed by an rSLDS can provide useful abstractions for planning and control. We present a novel hierarchical model-based algorithm inspired by Active Inference in which a discrete MDP sits above a low-level linear-quadratic controller. The recurrent transition dynamics learned by the rSLDS allow us to (1) specify temporally-abstracted sub-goals in a method reminiscent of the options framework, (2) lift the exploration into discrete space allowing us to exploit information-theoretic exploration bonuses and (3) `cache' the approximate solutions to low-level problems in the discrete planner. We successfully apply our model to the sparse Continuous Mountain Car task, demonstrating fast system identification via enhanced exploration and non-trivial planning through the delineation of abstract sub-goals.