# Q-Learning with Stable Infinite-Dimensional Linear Function Approximation

arXiv: https://arxiv.org/abs/2608.22636 · HTML: https://mlsanity.com/paper/2608.22636/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 3 of its 47 references that are in the map.
- First posted on arXiv 23 Aug 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 3 of 3)

- [Optimal variance-reduced stochastic approximation in Banach spaces](https://mlsanity.com/paper/2201.08518/) (2022)
- [Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence](https://mlsanity.com/paper/2603.03523/) (2026)
- [Periodic Regularized Q-Learning](https://mlsanity.com/paper/2602.03301/) (2026)

## Abstract (arXiv)

Q-learning with linear function approximation can be unstable because an arbitrary approximation architecture need not preserve the Bellman contraction. We develop a stable infinite-dimensional linear function approximation framework for Q-learning from a single Markovian behavior-policy trajectory. The learning variable is a coefficient field θ∈ C(mathbb L) on a compact latent metric space (mathbb L,ρ). The framework uses a reconstruction operator that maps θ to a continuous Q-function and a compression operator that maps Bellman updates back to latent coordinates. Nonexpansiveness of both operators induces a contractive latent Bellman map on C(mathbb L), with a unique fixed point θ^* whose reconstruction approximates the optimal Q-function up to representation error. We propose two stochastic approximation (SA) algorithms and establish their sup-norm convergence bounds with a leading term of order widetilde O(n-1/2). The infinite-dimensional formulation provides a powerful abstraction for identifying the structures that govern statistical difficulty. Smoothness of the compression map in ρ is inherited by θ^* and the SA iterates, allowing uniform estimation errors to be controlled through covering numbers of (mathbb L,ρ) rather than the dimension of C(mathbb L). Remarkably, the SA algorithms we propose are agnostic to the choice of ρ, and thus can automatically adapt to both the smoothness and the geometry. We further illustrate the framework through Q-measure-learning with linear density approximation and output-layer neural weight training under a frozen pretrained network.