# Universal Approximation Theorem of Deep Q-Networks

arXiv: https://arxiv.org/abs/2505.02288 · HTML: https://mlsanity.com/paper/2505.02288/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 4 May 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates](https://mlsanity.com/paper/2008.10546/) (2020)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients](https://mlsanity.com/paper/1809.07321/) (2018)
- [Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time](https://mlsanity.com/paper/1912.10697/) (2019)

## Built on by (most cited first, 6 of 6)

- [Universal Approximation Theorem for Deep Q-Learning via FBSDE System](https://mlsanity.com/paper/2505.06023/) (2025)
- [Optimal Depth of Neural Networks](https://mlsanity.com/paper/2506.16862/) (2025)
- [Neural Hamiltonian Operator](https://mlsanity.com/paper/2507.01313/) (2025)
- [Robust Control with Gradient Uncertainty](https://mlsanity.com/paper/2507.15082/) (2025)
- [Algorithms and data structures for automatic precision estimation of neural networks](https://mlsanity.com/paper/2509.24607/) (2025)
- [Deep Q-Learning on H"older Spaces](https://mlsanity.com/paper/2606.16846/) (2026)

## Abstract (arXiv)

We establish a continuous-time framework for analyzing Deep Q-Networks (DQNs) via stochastic control and Forward-Backward Stochastic Differential Equations (FBSDEs). Considering a continuous-time Markov Decision Process (MDP) driven by a square-integrable martingale, we analyze DQN approximation properties. We show that DQNs can approximate the optimal Q-function on compact sets with arbitrary accuracy and high probability, leveraging residual network approximation theorems and large deviation bounds for the state-action process. We then analyze the convergence of a general Q-learning algorithm for training DQNs in this setting, adapting stochastic approximation theorems. Our analysis emphasizes the interplay between DQN layer count, time discretization, and the role of viscosity solutions (primarily for the value function V^*) in addressing potential non-smoothness of the optimal Q-function. This work bridges deep reinforcement learning and stochastic control, offering insights into DQNs in continuous-time settings, relevant for applications with physical systems or high-frequency data.