Q-Learning with Stable Infinite-Dimensional Linear Function Approximation

Q-learning with linear function approximation can be unstable because an arbitrary approximation architecture need not preserve the Bellman contraction. We develop a stable infinite-dimensional linear function approximation framework for Q-learning from a single Markovian behavior-policy trajectory. The learning variable is a coefficient field θ∈ C(mathbb L) on a compact latent metric space (mathbb L,ρ). The framework uses a reconstruction operator that maps θ to a continuous Q-function and a compression operator that maps Bellman updates back to latent coordinates. Nonexpansiveness of both operators induces a contractive latent Bellman map on C(mathbb L), with a unique fixed point θ^* whose reconstruction approximates the optimal Q-function up to representation error. We propose two stochastic approximation (SA) algorithms and establish their sup-norm convergence bounds with a leading term of order widetilde O(n-1/2). The infinite-dimensional formulation provides a powerful abstraction for identifying the structures that govern statistical difficulty. Smoothness of the compression map in ρ is inherited by θ^* and the SA iterates, allowing uniform estimation errors to be controlled through covering numbers of (mathbb L,ρ) rather than the dimension of C(mathbb L). Remarkably, the SA algorithms we propose are agnostic to the choice of ρ, and thus can automatically adapt to both the smoothness and the geometry. We further illustrate the framework through Q-measure-learning with linear density approximation and output-layer neural weight training under a frozen pretrained network.