# Deconstructing deep active inference

arXiv: https://arxiv.org/abs/2303.01618 · HTML: https://mlsanity.com/paper/2303.01618/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [End-to-End Pixel-Based Deep Active Inference for Body Perception and Action](https://mlsanity.com/paper/2001.05847/) (2019)
- [Active Inference and Behavior Trees for Reactive Action Planning and Execution in Robotics](https://mlsanity.com/paper/2011.09756/) (2020)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [Branching Time Active Inference: the theory and its generality](https://mlsanity.com/paper/2111.11107/) (2021)
- [Robot self/other distinction: active inference meets neural networks learning in a mirror](https://mlsanity.com/paper/2004.05473/) (2020)
- [How do Variational Autoencoders Learn? Insights from Representational Similarity](https://mlsanity.com/paper/2205.08399/) (2022)
- [Branching Time Active Inference: empirical study and complexity class analysis](https://mlsanity.com/paper/2111.11276/) (2021)
- [A deep active inference model of the rubber-hand illusion](https://mlsanity.com/paper/2008.07408/) (2020)
- [Active inference body perception and action for humanoid robots](https://mlsanity.com/paper/1906.03022/) (2019)
- [Branching Time Active Inference with Bayesian Filtering](https://mlsanity.com/paper/2112.07406/) (2021)
- [Active Inference for Robotic Manipulation](https://mlsanity.com/paper/2206.10313/) (2022)
- [Multi-Modal and Multi-Factor Branching Time Active Inference](https://mlsanity.com/paper/2206.12503/) (2022)

## Built on by (most cited first, 5 of 5)

- [Structure learning with Temporal Gaussian Mixture for model-based Reinforcement Learning](https://mlsanity.com/paper/2411.11511/) (2024)
- [An Active Inference perspective on Neurofeedback Training](https://mlsanity.com/paper/2505.03308/) (2025)
- [Deep hybrid models: infer and plan in a dynamic world](https://mlsanity.com/paper/2402.10088/) (2024)
- [Dynamic planning in hierarchical active inference](https://mlsanity.com/paper/2402.11658/) (2024)
- [CogNarr Ecosystem: Facilitating Group Cognition at Scale](https://mlsanity.com/paper/2407.18945/) (2024)

## Abstract (arXiv)

Active inference is a theory of perception, learning and decision making, which can be applied to neuroscience, robotics, and machine learning. Recently, reasearch has been taking place to scale up this framework using Monte-Carlo tree search and deep learning. The goal of this activity is to solve more complicated tasks using deep active inference. First, we review the existing literature, then, we progresively build a deep active inference agent. For two agents, we have experimented with five definitions of the expected free energy and three different action selection strategies. According to our experiments, the models able to solve the dSprites environment are the ones that maximise rewards. Finally, we compare the similarity of the representation learned by the layers of various agents using centered kernel alignment. Importantly, the agent maximising reward and the agent minimising expected free energy learn very similar representations except for the last layer of the critic network (reflecting the difference in learning objective), and the variance layers of the transition and encoder networks. We found that the reward maximising agent is a lot more certain than the agent minimising expected free energy. This is because the agent minimising expected free energy always picks the action down, and does not gather enough data for the other actions. In contrast, the agent maximising reward, keeps on selecting the actions left and right, enabling it to successfully solve the task. The only difference between those two agents is the epistemic value, which aims to make the outputs of the transition and encoder networks as close as possible. Thus, the agent minimising expected free energy picks a single action (down), and becomes an expert at predicting the future when selecting this action. This makes the KL divergence between the output of the transition and encoder networks small.