mlsanityData to 7 Oct 2026

Generalised Bellman recurrence and three dualities in sequential decision-making

arXiv 2607.18077 ↗First posted 20 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the aggregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming return or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 75 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

25 of 75 references.

References · 25

  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  4. 2019A Theory of Regularized Markov Decision Processescited by 279
  5. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  6. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  7. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  8. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  9. 2020Active inference on discrete state-spaces: a synthesiscited by 91
  10. 2020Discovering Reinforcement Learning Algorithmscited by 75
  11. 2013Empowerment -- an Introductioncited by 65
  12. 2020Sophisticated Inferencecited by 56
13 more
  1. 2019A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statisticscited by 55
  2. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  3. 2020On the Relationship Between Active Inference and Control as Inferencecited by 42
  4. 2020Whence the Expected Free Energy?cited by 40
  5. 2021Regularized Policies are Reward Robustcited by 32
  6. 2015Information-Theoretic Bounded Rationalitycited by 21
  7. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  8. 2020Profunctor Optics, a Categorical Updatecited by 6
  9. 2022Value Iteration is Optic Compositioncited by 5
  10. 2019A Theoretical Connection Between Statistical Physics and Reinforcement Learningcited by 4
  11. 2024Reinforcement Learning in Categorical Cyberneticscited by 4
  12. 2025Recursive Reward Aggregationcited by 2
  13. 2016Memory shapes time perception and intertemporal choicescited by 1

Built on by

Not cited yet.