# Distributional Bellman Operators over Mean Embeddings

arXiv: https://arxiv.org/abs/2312.07358 · HTML: https://mlsanity.com/paper/2312.07358/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [An Analysis of Categorical Distributional Reinforcement Learning](https://mlsanity.com/paper/1802.08163/) (2018)
- [Statistics and Samples in Distributional Reinforcement Learning](https://mlsanity.com/paper/1902.08102/) (2019)
- [An Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2301.04462/) (2023)
- [Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping](https://mlsanity.com/paper/1910.02787/) (2019)
- [Distributional Reinforcement Learning for Multi-Dimensional Reward Functions](https://mlsanity.com/paper/2110.13578/) (2021)
- [Distributional Offline Policy Evaluation with Predictive Error Guarantees](https://mlsanity.com/paper/2302.09456/) (2023)
- [Hilbert Space Embeddings of Predictive State Representations](https://mlsanity.com/paper/1309.6819/) (2013)
- [Distributional Multivariate Policy Evaluation and Exploration with the Bellman GAN](https://mlsanity.com/paper/1808.01960/) (2018)
- [A neurally plausible model learns successor representations in partially observable environments](https://mlsanity.com/paper/1906.09480/) (2019)
- [Beyond Average Return in Markov Decision Processes](https://mlsanity.com/paper/2310.20266/) (2023)
- [Flexible and accurate inference and learning for deep generative models](https://mlsanity.com/paper/1805.11051/) (2018)
- [GAN Q-learning](https://mlsanity.com/paper/1805.04874/) (2018)
- [Coding with transient trajectories in recurrent neural networks](https://mlsanity.com/paper/1811.07592/) (2018)
- [Value Function Approximations via Kernel Embeddings for No-Regret Reinforcement Learning](https://mlsanity.com/paper/2011.07881/) (2020)

## Built on by (most cited first, 4 of 4)

- [A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.14172/) (2025)
- [Learning Counterfactual Distributions via Kernel Nearest Neighbors](https://mlsanity.com/paper/2410.13381/) (2024)
- [A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metric](https://mlsanity.com/paper/2603.12576/) (2026)
- [Distributional Soft Bellman Operator under the Cram'er Geometry](https://mlsanity.com/paper/2607.17897/) (2026)

## Abstract (arXiv)

We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework, provide asymptotic convergence theory, and examine the empirical performance of the algorithms on a suite of tabular tasks. Further, we show that this approach can be straightforwardly combined with deep reinforcement learning, and obtain a new deep RL agent that improves over baseline distributional approaches on the Arcade Learning Environment.