mlsanityData to 14 Oct 2026

Distributional Bellman Operators over Mean Embeddings

arXiv 2312.07358 ↗First posted 9 Dec 2023stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework, provide asymptotic convergence theory, and examine the empirical performance of the algorithms on a suite of tabular tasks. Further, we show that this approach can be straightforwardly combined with deep reinforcement learning, and obtain a new deep RL agent that improves over baseline distributional approaches on the Arcade Learning Environment.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 55 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

18 of 55 references.

References · 18

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  5. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  6. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  7. 2023An Analysis of Quantile Temporal-Difference Learningcited by 31
  8. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 29
  9. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 22
  10. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 20
  11. 2013Hilbert Space Embeddings of Predictive State Representationscited by 16
  12. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
6 more
  1. 2019A neurally plausible model learns successor representations in partially observable environmentscited by 12
  2. 2023Beyond Average Return in Markov Decision Processescited by 11
  3. 2018Flexible and accurate inference and learning for deep generative modelscited by 9
  4. 2018GAN Q-learningcited by 8
  5. 2018Coding with transient trajectories in recurrent neural networkscited by 8
  6. 2020Value Function Approximations via Kernel Embeddings for No-Regret Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 6
  2. 2024Learning Counterfactual Distributions via Kernel Nearest Neighborscited by 2
  3. 2026A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metriccited by 1
  4. 2026Distributional Soft Bellman Operator under the Cram'er Geometrycited by 0