mlsanityData to 14 Oct 2026

The Dually Flat Geometry of Planning as Inference

arXiv 2609.04005 ↗First posted 3 Sep 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We present an alternative characterization of the occupancy measure of reinforcement learning, obtained by embedding the planning criterion into the dynamics through a resetting planning process. Its stationary measure, which we term visitation measure, is the object on which the information geometry of decision making is most naturally expressed. The achievable visitation measures form a dually flat statistical manifold whose two affine charts are the visitation probabilities and the log-policies, dual under the conditional entropy. This structure makes planning-as-inference generalize from linear rewards to nonlinear functionals of the visitation, each iterate solved by one natural-gradient step, and gives the temporal-difference error the interpretation of a marginal-utility estimate. We develop the geometry and its consequences for reinforcement learning and theoretical neuroscience.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 70 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

36 of 70 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  8. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  10. 2018Maximum a Posteriori Policy Optimisationcited by 328
  11. 2019A Theory of Regularized Markov Decision Processescited by 286
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 242
24 more
  1. 2017A unified view of entropy-regularized Markov decision processescited by 182
  2. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  3. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  4. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  5. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  6. 2020Active inference on discrete state-spaces: a synthesiscited by 92
  7. 2021Reward is enough for convex MDPscited by 84
  8. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  9. 2013The Information Geometry of Mirror Descentcited by 54
  10. 2016Unifying task specification in reinforcement learningcited by 41
  11. 2020Whence the Expected Free Energy?cited by 40
  12. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  13. 2022Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EMcited by 34
  14. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  15. 2018Hessian Riemannian gradient flows in convex programmingcited by 26
  16. 2024CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learningcited by 23
  17. 2018Natural gradient via optimal transportcited by 20
  18. 2022Geometry and convergence of natural policy gradient methodscited by 12
  19. 2023Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descentcited by 6
  20. 2021Canonical Cortical Circuits and the Duality of Bayesian Inference and Optimal Controlcited by 2
  21. 2026Stochastic Decision Horizons for Survival-Based Constrained Reinforcement Learningcited by 2
  22. 2026Extended predictive coding framework as variational free-energy minimisation under exponential-family assumptioncited by 2
  23. 2026Linking PageRank, Time Reversal, and Policy Evaluationcited by 1
  24. 2026Active Inference as a Convex Markov Decision Processcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformation
  2. 2025Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces
  3. 2025The Geometry of Nonlinear Reinforcement Learning
  4. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measures
  5. 2026Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction
  6. 2024Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy
  7. 2026Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
  8. 2024Reinforcement Learning with Lookahead Information