mlsanityData to 14 Oct 2026

Active Inference as a Convex Markov Decision Process

arXiv 2607.20152 ↗First posted 22 Jul 2026cs.LG cs.AI stat.MLcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). This perspective reveals that policy-dependent reward prediction errors transmit natural gradients of the expected free energy backwards in time rather than up a hierarchy. Finally, we show that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning. Together this places EFE minimization within modern reinforcement learning and optimization theory and opens a route toward principled algorithms for active inference.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 52 of its 74 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

52 of 74 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Performative Policy Gradient: Optimality in Performative Reinforcement Learning23 Dec
  2. 2025Achieve Performatively Optimal Policy for Performative Reinforcement Learning6 Oct
  3. 2025Average-Reward Soft Actor-Critic15 Jan
  4. 2024Performative Reinforcement Learning with Linear Markov Decision Process7 Nov

Other references · 48

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  7. 2020Performative Predictioncited by 327
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  10. 2019Efficient Exploration via State Marginal Matchingcited by 188
  11. 2017A unified view of entropy-regularized Markov decision processescited by 182
  12. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
36 more
  1. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  2. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  3. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  4. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  5. 2021The Bayesian Learning Rulecited by 99
  6. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 96
  7. 2020Active inference on discrete state-spaces: a synthesiscited by 92
  8. 2021Reward is enough for convex MDPscited by 84
  9. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 73
  10. 2020Deep active inference agents using Monte-Carlo methodscited by 72
  11. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  12. 2020Sophisticated Inferencecited by 56
  13. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  14. 2013The Information Geometry of Mirror Descentcited by 54
  15. 2019Deep Active Inference as Variational Policy Gradientscited by 53
  16. 2023Performative Prediction: Past and Futurecited by 53
  17. 2020On the Relationship Between Active Inference and Control as Inferencecited by 42
  18. 2020Whence the Expected Free Energy?cited by 40
  19. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  20. 2022Performative Reinforcement Learningcited by 27
  21. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  22. 2021Contrastive Active Inferencecited by 21
  23. 2024Performative Reinforcement Learning in Gradually Shifting Environmentscited by 14
  24. 2021Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Controlcited by 10
  25. 2025Expected Free Energy-based Planning as Variational Inferencecited by 10
  26. 2020Active Inference or Control as Inference? A Unifying Viewcited by 9
  27. 2024What type of inference is planning?cited by 9
  28. 2023Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descentcited by 6
  29. 2024Reframing the Expected Free Energy: Four Formulations and a Unificationcited by 5
  30. 2024Value of Information and Reward Specification in Active Inference and POMDPscited by 4
  31. 2025Deep Active Inference Agents for Delayed and Long-Horizon Environmentscited by 4
  32. 2024The Number of Trials Matters in Infinite-Horizon General-Utility Markov Decision Processescited by 3
  33. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  34. 2026Extended predictive coding framework as variational free-energy minimisation under exponential-family assumptioncited by 2
  35. 2026What Type of Inference is Active Inference?cited by 2
  36. 2025The Geometry of Nonlinear Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Dually Flat Geometry of Planning as Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Bayesian policy selection using active inference
  2. 2021Prior Preference Learning from Experts:Designing a Reward with Active Inference
  3. 2024Demonstrating the Continual Learning Capabilities and Practical Application of Discrete-Time Active Inference
  4. 2021Online reinforcement learning with sparse rewards through an active inference capsule
  5. 2026Distributional Active Inference
  6. 2024Active Inference as a Model of Agency
  7. 2026Belief-Based Maximum Occupancy Principle and Active Inference
  8. 2025Boosting MCTS with Free Energy Minimization