mlsanityData to 14 Oct 2026

Deep Q-Learning on H"older Spaces

arXiv 2606.16846 ↗First posted 15 Jun 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We study the operator-theoretic core of Q-learning in continuous-time stochastic control with continuous states and actions. In value-based reinforcement learning, each Q-learning or DQN update is built from a Bellman optimality target; our analysis isolates this target in a diffusion setting and studies its regularity and approximation complexity. Under uniform ellipticity and H"older-regular coefficients, we show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable. This yields a compact family of Bellman iterates and motivates a tensor-product DeepONet architecture adapted to the mixed regularity of the problem. We then derive explicit approximation and resource bounds, together with a stiffness--complexity trade-off as the time step δ → 0. The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous stochastic control. At the same time, we do not claim a full convergence theorem for practical sampled Q-learning with exploration, replay, and stochastic gradient updates.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 23 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

11 of 23 references.

References · 11

  1. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  3. 2016Error bounds for approximations with deep ReLU networkscited by 574
  4. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  5. 2018Is Q-learning Provably Efficient?cited by 487
  6. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  7. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  8. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  9. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  10. 2017Neural networks and rational functionscited by 40
  11. 2025Universal Approximation Theorem of Deep Q-Networkscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-Learning with Stable Infinite-Dimensional Linear Function Approximation
  2. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
  3. 2025Universal Approximation Theorem for Deep Q-Learning via FBSDE System
  4. 2025Deep neural networks can provably solve Bellman equations for Markov decision processes without the curse of dimensionality
  5. 2021Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity
  6. 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
  7. 2024Regularized Q-Learning with Linear Function Approximation
  8. 2026Mixing Regularization for Average-Reward Q-Learning and Q-Planning