# Free Energy Projective Simulation (FEPS): Active inference with interpretability

arXiv: https://arxiv.org/abs/2411.14991 · HTML: https://mlsanity.com/paper/2411.14991/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 100 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 30 of 49)

- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Discovering and Achieving Goals via World Models](https://mlsanity.com/paper/2110.09514/) (2021)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Sophisticated Inference](https://mlsanity.com/paper/2006.04120/) (2020)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [Reinforcement Learning through Active Inference](https://mlsanity.com/paper/2002.12636/) (2020)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [Whence the Expected Free Energy?](https://mlsanity.com/paper/2004.08128/) (2020)
- [Scaling active inference](https://mlsanity.com/paper/1911.10601/) (2019)
- [Active Preference Inference using Language Models and Probabilistic Reasoning](https://mlsanity.com/paper/2312.12009/) (2023)
- [End-to-End Pixel-Based Deep Active Inference for Body Perception and Action](https://mlsanity.com/paper/2001.05847/) (2019)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Contrastive Active Inference](https://mlsanity.com/paper/2110.10083/) (2021)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [Learning Perception and Planning with Deep Active Inference](https://mlsanity.com/paper/2001.11841/) (2020)
- [On efficient computation in active inference](https://mlsanity.com/paper/2307.00504/) (2023)
- [Learning higher-order sequential structure with cloned HMMs](https://mlsanity.com/paper/1905.00507/) (2019)
- [Collective behavior from surprise minimization](https://mlsanity.com/paper/2307.14804/) (2023)
- [R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models](https://mlsanity.com/paper/2409.14216/) (2024)

## Built on by (most cited first, 2 of 2)

- [Information as Maximum-Caliber Deviation: A bridge between Integrated Information Theory and the Free Energy Principle](https://mlsanity.com/paper/2605.12536/) (2026)
- [Interpreting Quantum Learning Models via Stochastic Processes](https://mlsanity.com/paper/2607.17327/) (2026)

## Abstract (arXiv)

In the last decade, the free energy principle (FEP) and active inference (AIF) have achieved many successes connecting conceptual models of learning and cognition to mathematical models of perception and action. This effort is driven by a multidisciplinary interest in understanding aspects of self-organizing complex adaptive systems, including elements of agency. Various reinforcement learning (RL) models performing active inference have been proposed and trained on standard RL tasks using deep neural networks. Recent work has focused on improving such agents' performance in complex environments by incorporating the latest machine learning techniques. In this paper, we take an alternative approach. Within the constraints imposed by the FEP and AIF, we attempt to model agents in an interpretable way without deep neural networks by introducing Free Energy Projective Simulation (FEPS). Using internal rewards only, FEPS agents build a representation of their partially observable environments with which they interact. Following AIF, the policy to achieve a given task is derived from this world model by minimizing the expected free energy. Leveraging the interpretability of the model, techniques are introduced to deal with long-term goals and reduce prediction errors caused by erroneous hidden state estimation. We test the FEPS model on two RL environments inspired from behavioral biology: a timed response task and a navigation task in a partially observable grid. Our results show that FEPS agents fully resolve the ambiguity of both environments by appropriately contextualizing their observations based on prediction accuracy only. In addition, they infer optimal policies flexibly for any target observation in the environment.