# Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filtering

arXiv: https://arxiv.org/abs/2403.20020 · HTML: https://mlsanity.com/paper/2403.20020/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 77 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Robust Reinforcement Learning using Offline Data](https://mlsanity.com/paper/2208.05129/) (2022)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces](https://mlsanity.com/paper/1405.6757/) (2014)
- [Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning with Polynomial Sample Complexity](https://mlsanity.com/paper/2006.03976/) (2020)
- [Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning](https://mlsanity.com/paper/2210.11317/) (2022)
- [Fej'er-monotone hybrid steepest descent method for affinely constrained and composite convex minimization tasks](https://mlsanity.com/paper/1608.02500/) (2016)

## Built on by (most cited first, 8 of 8)

- [Online reinforcement learning via sparse Gaussian mixture model Q-functions](https://mlsanity.com/paper/2509.14585/) (2025)
- [Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learning](https://mlsanity.com/paper/2503.16192/) (2025)
- [Nonconvex Regularization for Feature Selection in Reinforcement Learning](https://mlsanity.com/paper/2509.15652/) (2025)
- [Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning](https://mlsanity.com/paper/2512.18763/) (2025)
- [Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization](https://mlsanity.com/paper/2409.04374/) (2024)
- [Non-Convex Sparse Reinforcement Learning via Non-Monotone Inclusions](https://mlsanity.com/paper/2607.04990/) (2026)
- [Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning](https://mlsanity.com/paper/2607.23474/) (2026)
- [A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning](https://mlsanity.com/paper/2609.29961/) (2026)

## Abstract (arXiv)

This paper designs novel nonparametric Bellman mappings in reproducing kernel Hilbert spaces (RKHSs) for reinforcement learning (RL). The proposed mappings benefit from the rich approximating properties of RKHSs, adopt no assumptions on the statistics of the data owing to their nonparametric nature, require no knowledge on transition probabilities of Markov decision processes, and may operate without any training data. Moreover, they allow for sampling on-the-fly via the design of trajectory samples, re-use past test data via experience replay, effect dimensionality reduction by random Fourier features, and enable computationally lightweight operations to fit into efficient online or time-adaptive learning. The paper offers also a variational framework to design the free parameters of the proposed Bellman mappings, and shows that appropriate choices of those parameters yield several popular Bellman-mapping designs. As an application, the proposed mappings are employed to offer a novel solution to the problem of countering outliers in adaptive filtering. More specifically, with no prior information on the statistics of the outliers and no training data, a policy-iteration algorithm is introduced to select online, per time instance, the ``optimal'' coefficient p in the least-mean-p-power-error method. Numerical tests on synthetic data showcase, in most of the cases, the superior performance of the proposed solution over several RL and non-RL schemes.