mlsanityData to 14 Oct 2026

Universal Approximation Theorem of Deep Q-Networks

arXiv 2505.02288 ↗First posted 4 May 2025cs.LG cs.AI stat.MLcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We establish a continuous-time framework for analyzing Deep Q-Networks (DQNs) via stochastic control and Forward-Backward Stochastic Differential Equations (FBSDEs). Considering a continuous-time Markov Decision Process (MDP) driven by a square-integrable martingale, we analyze DQN approximation properties. We show that DQNs can approximate the optimal Q-function on compact sets with arbitrary accuracy and high probability, leveraging residual network approximation theorems and large deviation bounds for the state-action process. We then analyze the convergence of a general Q-learning algorithm for training DQNs in this setting, adapting stochastic approximation theorems. Our analysis emphasizes the interplay between DQN layer count, time discretization, and the role of viscosity solutions (primarily for the value function V^*) in addressing potential non-smoothness of the optimal Q-function. This work bridges deep reinforcement learning and stochastic control, offering insights into DQNs in continuous-time settings, relevant for applications with physical systems or high-frequency data.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 38 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

11 of 38 references.

References · 11

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2016Error bounds for approximations with deep ReLU networkscited by 574
  5. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  6. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 179
  7. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  8. 2020SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimatescited by 75
  9. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  10. 2018A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficientscited by 31
  11. 2019Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Timecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Universal Approximation Theorem for Deep Q-Learning via FBSDE Systemcited by 2
  2. 2025Optimal Depth of Neural Networkscited by 0
  3. 2025Neural Hamiltonian Operatorcited by 0
  4. 2025Robust Control with Gradient Uncertaintycited by 0
  5. 2025Algorithms and data structures for automatic precision estimation of neural networkscited by 0
  6. 2026Deep Q-Learning on H"older Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Approximation to Deep Q-Network by Stochastic Delay Differential Equations
  2. 2025Deep neural networks can provably solve Bellman equations for Markov decision processes without the curse of dimensionality
  3. 2026Spectral Analysis of Dueling Q-Learning
  4. 2025Beyond ReLU: Chebyshev-DQN for Enhanced Deep Q-Networks
  5. 2024Stochastic Q-learning for Large Discrete Action Spaces
  6. 2025GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
  7. 2025DQN Performance with Epsilon Greedy Policies and Prioritized Experience Replay
  8. 2024VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability