# Deep Active Inference as Variational Policy Gradients

arXiv: https://arxiv.org/abs/1907.03876 · HTML: https://mlsanity.com/paper/1907.03876/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2019.

## Built on by (most cited first, 30 of 53)

- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [pymdp: A Python library for active inference in discrete state spaces](https://mlsanity.com/paper/2201.03904/) (2022)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Contrastive Active Inference](https://mlsanity.com/paper/2110.10083/) (2021)
- [Branching Time Active Inference: the theory and its generality](https://mlsanity.com/paper/2111.11107/) (2021)
- [An empirical evaluation of active inference in multi-armed bandits](https://mlsanity.com/paper/2101.08699/) (2021)
- [Backprop-Free Reinforcement Learning with Active Neural Generative Coding](https://mlsanity.com/paper/2107.07046/) (2021)
- [Active Inference as a Model of Agency](https://mlsanity.com/paper/2401.12917/) (2024)
- [Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control](https://mlsanity.com/paper/2103.06859/) (2021)
- [R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models](https://mlsanity.com/paper/2409.14216/) (2024)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Branching Time Active Inference: empirical study and complexity class analysis](https://mlsanity.com/paper/2111.11276/) (2021)
- [Hierarchical generative modelling for autonomous robots](https://mlsanity.com/paper/2308.07775/) (2023)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [Prior Preference Learning from Experts:Designing a Reward with Active Inference](https://mlsanity.com/paper/2101.08937/) (2021)
- [Active Inference and Epistemic Value in Graphical Models](https://mlsanity.com/paper/2109.00541/) (2021)
- [Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems](https://mlsanity.com/paper/2209.09174/) (2022)
- [Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents](https://mlsanity.com/paper/2203.10592/) (2022)
- [Deconstructing deep active inference](https://mlsanity.com/paper/2303.01618/) (2023)
- [Habits and goals in synergy: a variational Bayesian framework for behavior](https://mlsanity.com/paper/2304.05008/) (2023)
- [Value of Information and Reward Specification in Active Inference and POMDPs](https://mlsanity.com/paper/2408.06542/) (2024)
- [Online reinforcement learning with sparse rewards through an active inference capsule](https://mlsanity.com/paper/2106.02390/) (2021)
- [Social Neuro AI: Social Interaction as the "dark matter" of AI](https://mlsanity.com/paper/2112.15459/) (2021)
- [Successor Representation Active Inference](https://mlsanity.com/paper/2207.09897/) (2022)
- [Learning Policies for Continuous Control via Transition Models](https://mlsanity.com/paper/2209.08033/) (2022)
- [Active Inference Tree Search in Large POMDPs](https://mlsanity.com/paper/2103.13860/) (2021)
- [Anomaly Detection via Controlled Sensing and Deep Active Inference](https://mlsanity.com/paper/2105.06288/) (2021)
- [Goal-Directed Planning by Reinforcement Learning and Active Inference](https://mlsanity.com/paper/2106.09938/) (2021)
- [Deep Active Inference for Pixel-Based Discrete Control: Evaluation on the Car Racing Problem](https://mlsanity.com/paper/2109.04155/) (2021)
- [A Neural Active Inference Model of Perceptual-Motor Learning](https://mlsanity.com/paper/2211.10419/) (2022)

## Abstract (arXiv)

Active Inference is a theory of action arising from neuroscience which casts action and planning as a bayesian inference problem to be solved by minimizing a single quantity - the variational free energy. Active Inference promises a unifying account of action and perception coupled with a biologically plausible process theory. Despite these potential advantages, current implementations of Active Inference can only handle small, discrete policy and state-spaces and typically require the environmental dynamics to be known. In this paper we propose a novel deep Active Inference algorithm which approximates key densities using deep neural networks as flexible function approximators, which enables Active Inference to scale to significantly larger and more complex tasks. We demonstrate our approach on a suite of OpenAIGym benchmark tasks and obtain performance comparable with common reinforcement learning baselines. Moreover, our algorithm shows similarities with maximum entropy reinforcement learning and the policy gradients algorithm, which reveals interesting connections between the Active Inference framework and reinforcement learning.