# Prior preferences in active inference agents: soft, hard, and goal shaping

arXiv: https://arxiv.org/abs/2512.03293 · HTML: https://mlsanity.com/paper/2512.03293/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 19 of 19)

- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [Sophisticated Inference](https://mlsanity.com/paper/2006.04120/) (2020)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [Reinforcement Learning through Active Inference](https://mlsanity.com/paper/2002.12636/) (2020)
- [Whence the Expected Free Energy?](https://mlsanity.com/paper/2004.08128/) (2020)
- [pymdp: A Python library for active inference in discrete state spaces](https://mlsanity.com/paper/2201.03904/) (2022)
- [Scaling active inference](https://mlsanity.com/paper/1911.10601/) (2019)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [Bayesian policy selection using active inference](https://mlsanity.com/paper/1904.08149/) (2019)
- [Exploration and preference satisfaction trade-off in reward-free learning](https://mlsanity.com/paper/2106.04316/) (2021)
- [Prior Preference Learning from Experts:Designing a Reward with Active Inference](https://mlsanity.com/paper/2101.08937/) (2021)
- [The Two Kinds of Free Energy and the Bayesian Revolution](https://mlsanity.com/paper/2004.11763/) (2020)
- [Active inference for action-unaware agents](https://mlsanity.com/paper/2508.12027/) (2025)
- [A coalgebraic perspective on predictive processing](https://mlsanity.com/paper/2508.16877/) (2025)

## Built on by (most cited first, 1 of 1)

- [Two Routes to Interoceptive Prioritisation: Dissociating Likelihood Precision and Preference Reweighting in Active Inference](https://mlsanity.com/paper/2610.18617/) (2026)

## Abstract (arXiv)

Active inference proposes expected free energy as an objective for planning and decision-making to adequately balance exploitative and explorative drives in learning agents. The exploitative drive, or what an agent wants to achieve, is formalised as the Kullback-Leibler divergence between a variational probability distribution, updated at each inference step, and a preference probability distribution that indicates what states or observations are more likely for the agent, hence determining the agent's goal in a certain environment. In the literature, the questions of how the preference distribution should be specified and of how a certain specification impacts inference and learning in an active inference agent have been given hardly any attention. In this work, we consider four possible ways of defining the preference distribution, either providing the agents with hard or soft goals and either involving or not goal shaping (i.e., intermediate goals). We compare the performances of four agents, each given one of the possible preference distributions, in a grid world navigation task. Our results show that goal shaping enables the best performance overall (i.e., it promotes exploitation) while sacrificing learning about the environment's transition dynamics (i.e., it hampers exploration).