# Hilbert Space Embeddings of Predictive State Representations

arXiv: https://arxiv.org/abs/1309.6819 · HTML: https://mlsanity.com/paper/1309.6819/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2013.

## Built on by (most cited first, 16 of 16)

- [PAC Reinforcement Learning for Predictive State Representations](https://mlsanity.com/paper/2207.05738/) (2022)
- [Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems](https://mlsanity.com/paper/2206.12020/) (2022)
- [Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithms](https://mlsanity.com/paper/2209.14990/) (2022)
- [Future-Dependent Value-Based Off-Policy Evaluation in POMDPs](https://mlsanity.com/paper/2207.13081/) (2022)
- [Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings](https://mlsanity.com/paper/2206.12081/) (2022)
- [Instrumental Variables in Causal Inference and Machine Learning: A Survey](https://mlsanity.com/paper/2212.05778/) (2022)
- [Topology, Convergence, and Reconstruction of Predictive States](https://mlsanity.com/paper/2109.09203/) (2021)
- [A Mathematical Characterization of Minimally Sufficient Robot Brains](https://mlsanity.com/paper/2308.09041/) (2023)
- [Distributional Bellman Operators over Mean Embeddings](https://mlsanity.com/paper/2312.07358/) (2023)
- [Coefficient-based Regularized Distribution Regression](https://mlsanity.com/paper/2208.12427/) (2022)
- [Sequence Modeling with Spectral Mean Flows](https://mlsanity.com/paper/2510.15366/) (2025)
- [Importance Weighting Approach in Kernel Bayes' Rule](https://mlsanity.com/paper/2202.02474/) (2022)
- [Exploring Predictive States via Cantor Embeddings and Wasserstein Distance](https://mlsanity.com/paper/2206.04198/) (2022)
- [Safe Reinforcement Learning in Tensor Reproducing Kernel Hilbert Space](https://mlsanity.com/paper/2312.00727/) (2023)
- [Learning Nonlinear Regime Transitions via Semi-Parametric State-Space Models](https://mlsanity.com/paper/2604.04963/) (2026)
- [Generative sequence modeling for infinite memory processes via predictive states](https://mlsanity.com/paper/2609.38524/) (2026)

## Abstract (arXiv)

Predictive State Representations (PSRs) are an expressive class of models for controlled stochastic processes. PSRs represent state as a set of predictions of future observable events. Because PSRs are defined entirely in terms of observable data, statistically consistent estimates of PSR parameters can be learned efficiently by manipulating moments of observed training data. Most learning algorithms for PSRs have assumed that actions and observations are finite with low cardinality. In this paper, we generalize PSRs to infinite sets of observations and actions, using the recent concept of Hilbert space embeddings of distributions. The essence is to represent the state as a nonparametric conditional embedding operator in a Reproducing Kernel Hilbert Space (RKHS) and leverage recent work in kernel methods to estimate, predict, and update the representation. We show that these Hilbert space embeddings of PSRs are able to gracefully handle continuous actions and observations, and that our learned models outperform competing system identification algorithms on several prediction benchmarks.