# On the connection between Bregman divergence and value in regularized Markov decision processes

arXiv: https://arxiv.org/abs/2210.12160 · HTML: https://mlsanity.com/paper/2210.12160/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 4 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 2 of 2)

- [On the Global Convergence Rates of Softmax Policy Gradient Methods](https://mlsanity.com/paper/2005.06392/) (2020)
- [Making Sense of Reinforcement Learning and Probabilistic Inference](https://mlsanity.com/paper/2001.00805/) (2020)

## Built on by (most cited first, 2 of 2)

- [Probabilistic Inference in Reinforcement Learning Done Right](https://mlsanity.com/paper/2311.13294/) (2023)
- [Efficient Exploration via Epistemic-Risk-Seeking Policy Optimization](https://mlsanity.com/paper/2302.09339/) (2023)

## Abstract (arXiv)

In this short note we derive a relationship between the Bregman divergence from the current policy to the optimal policy and the suboptimality of the current value function in a regularized Markov decision process. This result has implications for multi-task reinforcement learning, offline reinforcement learning, and regret analysis under function approximation, among others.