# Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filtering

arXiv: https://arxiv.org/abs/2309.07548 · HTML: https://mlsanity.com/paper/2309.07548/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 38 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 4 of 4)

- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces](https://mlsanity.com/paper/1405.6757/) (2014)
- [Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning](https://mlsanity.com/paper/2210.11317/) (2022)

## Built on by (most cited first, 4 of 4)

- [Online reinforcement learning via sparse Gaussian mixture model Q-functions](https://mlsanity.com/paper/2509.14585/) (2025)
- [Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning](https://mlsanity.com/paper/2512.18763/) (2025)
- [Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization](https://mlsanity.com/paper/2409.04374/) (2024)
- [Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning](https://mlsanity.com/paper/2607.23474/) (2026)

## Abstract (arXiv)

This paper aims at the algorithmic/theoretical core of reinforcement learning (RL) by introducing the novel class of proximal Bellman mappings. These mappings are defined in reproducing kernel Hilbert spaces (RKHSs), to benefit from the rich approximation properties and inner product of RKHSs, they are shown to belong to the powerful Hilbertian family of (firmly) nonexpansive mappings, regardless of the values of their discount factors, and possess ample degrees of design freedom to even reproduce attributes of the classical Bellman mappings and to pave the way for novel RL designs. An approximate policy-iteration scheme is built on the proposed class of mappings to solve the problem of selecting online, at every time instance, the "optimal" exponent p in a p-norm loss to combat outliers in linear adaptive filtering, without training data and any knowledge on the statistical properties of the outliers. Numerical tests on synthetic data showcase the superior performance of the proposed framework over several non-RL and kernel-based RL schemes.