# Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach

arXiv: https://arxiv.org/abs/2601.18952 · HTML: https://mlsanity.com/paper/2601.18952/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 26 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Abstract (arXiv)

We propose Kernel Embedding Distributional Reinforcement Learning (KE-DRL) an offline method to estimate kernel mean embeddings of conditional multivariate return distributions from observed offline trajectories with continuous state-action inputs and vector-valued rewards. KE-DRL approximates the conditional embedding and estimates its a finite-dictionary representation coefficients through a maximum mean discrepancy Bellman criterion. For a regular class of return distributions, we show that a Mat'ern embedding and a Bellman-invariant Sobolev-moment class identifies the unique distributional Bellman fixed point and and derive a H"older-type bound that controls Wasserstein error by the population embedding Bellman residual. For directly observed responses, we provide finite-sample and uniform error bounds for a regularized conditional mean embedding estimator. Simulation experiments evaluate pointwise embedding recovery against Monte Carlo benchmarks across multiple behavior-target policy pairs. An application to Expedia hotel-search data illustrates conditional multivariate return evaluation under two policies.