# Yet Another Distributional Bellman Equation

arXiv: https://arxiv.org/abs/2505.21098 · HTML: https://mlsanity.com/paper/2505.21098/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2025.

## Built on by (most cited first, 2 of 2)

- [Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity](https://mlsanity.com/paper/2602.03778/) (2026)
- [Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent](https://mlsanity.com/paper/2602.06567/) (2026)

## Abstract (arXiv)

We consider non-standard Markov Decision Processes (MDPs) where the target function is not only a simple expectation of the accumulated reward. Instead, we consider rather general functionals of the joint distribution of terminal state and accumulated reward which have to be optimized. For finite state and compact action space, we show how to solve these problems by defining a lifted MDP whose state space is the space of distributions over the true states of the process. We derive a Bellman equation in this setting, which can be considered as a distributional Bellman equation. Well-known cases like the standard MDP and quantile MDPs are shown to be special examples of our framework. We also apply our model to a variant of an optimal transport problem.