mlsanityData to 14 Oct 2026

Value of Information and Reward Specification in Active Inference and POMDPs

arXiv 2408.06542 ↗First posted 13 Aug 2024cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Expected free energy (EFE) is a central quantity in active inference which has recently gained popularity due to its intuitive decomposition of the expected value of control into a pragmatic and an epistemic component. While numerous conjectures have been made to justify EFE as a decision making objective function, the most widely accepted is still its intuitiveness and resemblance to variational free energy in approximate Bayesian inference. In this work, we take a bottom up approach and ask: taking EFE as given, what's the resulting agent's optimality gap compared with a reward-driven reinforcement learning (RL) agent, which is well understood? By casting EFE under a particular class of belief MDP and using analysis tools from RL theory, we show that EFE approximates the Bayes optimal RL policy via information value. We discuss the implications for objective specification of active inference agents.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 63 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

31 of 63 references.

References · 31

  1. 2016Variational Inference: A Review for Statisticianscited by 1,472
  2. 2018MINE: Mutual Information Neural Estimationcited by 683
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  4. 2017VAE with a VampPriorcited by 248
  5. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  6. 2019Efficient Exploration via State Marginal Matchingcited by 188
  7. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  8. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  9. 2020Active inference on discrete state-spaces: a synthesiscited by 92
  10. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  11. 2021Reinforcement Learning, Bit by Bitcited by 59
  12. 2020Sophisticated Inferencecited by 56
19 more
  1. 2019Deep Active Inference as Variational Policy Gradientscited by 53
  2. 2021Active Inference in Robotics and Artificial Agents: Survey and Challengescited by 52
  3. 2020On the Relationship Between Active Inference and Control as Inferencecited by 42
  4. 2020Action and Perception as Divergence Minimizationcited by 32
  5. 2019Scaling active inferencecited by 30
  6. 2022The free energy principle made simpler but not too simplecited by 27
  7. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  8. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  9. 2022Path integrals, particular kinds, and strange thingscited by 20
  10. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  11. 2024Active Inference as a Model of Agencycited by 12
  12. 2020Active Inference or Control as Inference? A Unifying Viewcited by 9
  13. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  14. 2020The Two Kinds of Free Energy and the Bayesian Revolutioncited by 7
  15. 2023Resolving uncertainty on the fly: Modeling adaptive driving behavior as active inferencecited by 6
  16. 2022Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agentscited by 5
  17. 2024Reframing the Expected Free Energy: Four Formulations and a Unificationcited by 5
  18. 2023STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learningcited by 4
  19. 2022Active Inference and Reinforcement Learning: A unified inference on continuous state and action spaces under partial observabilitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2024Possible Principles for Aligned Structure Learning Agentscited by 3
  2. 2024Free Energy Projective Simulation (FEPS): Active inference with interpretabilitycited by 2
  3. 2026Active Inference as a Convex Markov Decision Processcited by 1
  4. 2026Leveraging the Value of Information in POMDP Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Prior Preference Learning from Experts:Designing a Reward with Active Inference
  2. 2019Bayesian policy selection using active inference
  3. 2026What Information Buys in Active Inference
  4. 2026Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators
  5. 2024Demonstrating the Continual Learning Capabilities and Practical Application of Discrete-Time Active Inference
  6. 2025Boosting MCTS with Free Energy Minimization
  7. 2026Belief-Based Maximum Occupancy Principle and Active Inference
  8. 2021Reinforced Imitation Learning by Free Energy Principle