# Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert Spaces

arXiv: https://arxiv.org/abs/1807.11274 · HTML: https://mlsanity.com/paper/1807.11274/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2018.

## Built on by (most cited first, 7 of 7)

- [On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control](https://mlsanity.com/paper/2106.08414/) (2021)
- [Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective](https://mlsanity.com/paper/2312.11214/) (2023)
- [Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning](https://mlsanity.com/paper/2501.04879/) (2025)
- [Policy Newton Algorithm in Reproducing Kernel Hilbert Space](https://mlsanity.com/paper/2506.01597/) (2025)
- [Towards Safe Continuing Task Reinforcement Learning](https://mlsanity.com/paper/2102.12585/) (2021)
- [Non-Parametric Stochastic Policy Gradient with Strategic Retreat for Non-Stationary Environment](https://mlsanity.com/paper/2203.14905/) (2022)
- [Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs](https://mlsanity.com/paper/2408.10015/) (2024)

## Abstract (arXiv)

Reinforcement learning consists of finding policies that maximize an expected cumulative long-term reward in a Markov decision process with unknown transition probabilities and instantaneous rewards. In this paper, we consider the problem of finding such optimal policies while assuming they are continuous functions belonging to a reproducing kernel Hilbert space (RKHS). To learn the optimal policy we introduce a stochastic policy gradient ascent algorithm with three unique novel features: (i) The stochastic estimates of policy gradients are unbiased. (ii) The variance of stochastic gradients is reduced by drawing on ideas from numerical differentiation. (iii) Policy complexity is controlled using sparse RKHS representations. Novel feature (i) is instrumental in proving convergence to a stationary point of the expected cumulative reward. Novel feature (ii) facilitates reasonable convergence times. Novel feature (iii) is a necessity in practical implementations which we show can be done in a way that does not eliminate convergence guarantees. Numerical examples in standard problems illustrate successful learning of policies with low complexity representations which are close to stationary points of the expected cumulative reward.