# Expected Free Energy as Belief-Dependent Utility for rho-POMDPs

arXiv: https://arxiv.org/abs/2607.16981 · HTML: https://mlsanity.com/paper/2607.16981/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 18 Jul 2026.

## Builds on (most cited first, 17 of 17)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Learning to Optimize via Information-Directed Sampling](https://mlsanity.com/paper/1403.5556/) (2014)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [Online algorithms for POMDPs with continuous state, action, and observation spaces](https://mlsanity.com/paper/1709.06196/) (2017)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Sophisticated Inference](https://mlsanity.com/paper/2006.04120/) (2020)
- [Reinforcement Learning through Active Inference](https://mlsanity.com/paper/2002.12636/) (2020)
- [On the Relationship Between Active Inference and Control as Inference](https://mlsanity.com/paper/2006.12964/) (2020)
- [Whence the Expected Free Energy?](https://mlsanity.com/paper/2004.08128/) (2020)
- [pymdp: A Python library for active inference in discrete state spaces](https://mlsanity.com/paper/2201.03904/) (2022)
- [Expected Free Energy-based Planning as Variational Inference](https://mlsanity.com/paper/2504.14898/) (2025)
- [Reframing the Expected Free Energy: Four Formulations and a Unification](https://mlsanity.com/paper/2402.14460/) (2024)
- [Active Inference Tree Search in Large POMDPs](https://mlsanity.com/paper/2103.13860/) (2021)

## Abstract (arXiv)

An agent acting under partial observability must decide when to gather information and which observations are worth their cost. Standard POMDPs value information only through its eventual effect on reward. The ρ-POMDP framework instead rewards uncertainty reduction directly, through a belief-dependent utility ρ, but in practice both the choice of ρ and the weight placed on it are tuned by hand for every task. We show that active inference removes this tuning entirely. Minimizing Expected Free Energy (EFE) is exactly equivalent to solving a ρ-POMDP whose utility is expected information gain, and the exploration weight is fixed at w=1 because the variational bound expresses pragmatic and epistemic value in the same units (nats). We prove this equivalence for observe-then-commit POMDPs and extend it to factored observation POMDPs, a broader class that covers interleaved observe-act problems such as non-destructive testing and mobile sensing, where gathering information leaves the hidden state unchanged. Experiments support the theory. Across environments ranging from the classic Tiger problem to RockSample and a new Structural Inspection benchmark with over 65,000 states, the untuned weight matches or outperforms reward-only planning at the same horizon, avoids the over-exploration of bonuses tuned per task, and sits near the reward-maximizing knee of the success-reward Pareto frontier. The practical payoff is an exploration objective that works out of the box. In applications such as fault detection and medical screening, where every test has a price and every missed fault has a cost, EFE supplies a belief-dependent utility that is derived rather than tuned.