# A Unified Bellman Equation for Causal Information and Value in Markov Decision Processes

arXiv: https://arxiv.org/abs/1703.01585 · HTML: https://mlsanity.com/paper/1703.01585/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2017.

## Built on by (most cited first, 11 of 11)

- [An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2103.06224/) (2021)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learning](https://mlsanity.com/paper/2210.16877/) (2022)
- [Bayesian Reinforcement Learning with Limited Cognitive Load](https://mlsanity.com/paper/2305.03263/) (2023)
- [GEM: Group Enhanced Model for Learning Dynamical Control Systems](https://mlsanity.com/paper/2104.02844/) (2021)
- [Data-Driven Optimization of Directed Information over Discrete Alphabets](https://mlsanity.com/paper/2301.00621/) (2023)
- [Variational Inference for Model-Free and Model-Based Reinforcement Learning](https://mlsanity.com/paper/2209.01693/) (2022)
- [Maximum Entropy Exploration Without the Rollouts](https://mlsanity.com/paper/2603.12325/) (2026)
- [Support sufficiency as action-sufficient compression: a single-cycle rate-regret formulation](https://mlsanity.com/paper/2606.09858/) (2026)
- [Discovering Intrinsic Reward with Contrastive Random Walk](https://mlsanity.com/paper/2204.10976/) (2022)
- [A Bellman Optimality Equation for Plasticity](https://mlsanity.com/paper/2609.10776/) (2026)

## Abstract (arXiv)

The interaction between an artificial agent and its environment is bi-directional. The agent extracts relevant information from the environment, and affects the environment by its actions in return to accumulate high expected reward. Standard reinforcement learning (RL) deals with the expected reward maximization. However, there are always information-theoretic limitations that restrict the expected reward, which are not properly considered by the standard RL. In this work we consider RL objectives with information-theoretic limitations. For the first time we derive a Bellman-type recursive equa- tion for the causal information between the environment and the agent, which is combined plausibly with the Bellman recursion for the value function. The unified equitation serves to explore the typical behavior of artificial agents in an infinite time horizon.