# Structure learning with Temporal Gaussian Mixture for model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2411.11511 · HTML: https://mlsanity.com/paper/2411.11511/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 35 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 10 of 10)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [A review of mean-shift algorithms for clustering](https://mlsanity.com/paper/1503.00687/) (2015)
- [Bayesian model reduction](https://mlsanity.com/paper/1805.07092/) (2018)
- [Robot self/other distinction: active inference meets neural networks learning in a mirror](https://mlsanity.com/paper/2004.05473/) (2020)
- [Deconstructing deep active inference](https://mlsanity.com/paper/2303.01618/) (2023)
- [Active inference body perception and action for humanoid robots](https://mlsanity.com/paper/1906.03022/) (2019)
- [Multi-Modal and Multi-Factor Branching Time Active Inference](https://mlsanity.com/paper/2206.12503/) (2022)

## Built on by (most cited first, 1 of 1)

- [AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models](https://mlsanity.com/paper/2505.24784/) (2025)

## Abstract (arXiv)

Model-based reinforcement learning refers to a set of approaches capable of sample-efficient decision making, which create an explicit model of the environment. This model can subsequently be used for learning optimal policies. In this paper, we propose a temporal Gaussian Mixture Model composed of a perception model and a transition model. The perception model extracts discrete (latent) states from continuous observations using a variational Gaussian mixture likelihood. Importantly, our model constantly monitors the collected data searching for new Gaussian components, i.e., the perception model performs a form of structure learning (Smith et al., 2020; Friston et al., 2018; Neacsu et al., 2022) as it learns the number of Gaussian components in the mixture. Additionally, the transition model learns the temporal transition between consecutive time steps by taking advantage of the Dirichlet-categorical conjugacy. Both the perception and transition models are able to forget part of the data points, while integrating the information they provide within the prior, which ensure fast variational inference. Finally, decision making is performed with a variant of Q-learning which is able to learn Q-values from beliefs over states. Empirically, we have demonstrated the model's ability to learn the structure of several mazes: the model discovered the number of states and the transition probabilities between these states. Moreover, using its learned Q-values, the agent was able to successfully navigate from the starting position to the maze's exit.