# Asymptotics of Reinforcement Learning with Neural Networks

arXiv: https://arxiv.org/abs/1911.07304 · HTML: https://mlsanity.com/paper/1911.07304/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2019.

## Built on by (most cited first, 10 of 10)

- [Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime](https://mlsanity.com/paper/2201.07296/) (2022)
- [Sample Complexity and Overparameterization Bounds for Temporal Difference Learning with Neural Network Approximation](https://mlsanity.com/paper/2103.01391/) (2021)
- [Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equations](https://mlsanity.com/paper/2305.06000/) (2023)
- [Normalization effects on deep neural networks](https://mlsanity.com/paper/2209.01018/) (2022)
- [Towards a Better Understanding of Representation Dynamics under TD-learning](https://mlsanity.com/paper/2305.18491/) (2023)
- [Global Convergence of the ODE Limit for Online Actor-Critic Algorithms in Reinforcement Learning](https://mlsanity.com/paper/2108.08655/) (2021)
- [Regularized Q-Learning with Linear Function Approximation](https://mlsanity.com/paper/2401.15196/) (2024)
- [Provable Watermarking for Data Poisoning Attacks](https://mlsanity.com/paper/2510.09210/) (2025)
- [Global Optimality of Elman-type RNN in the Mean-Field Regime](https://mlsanity.com/paper/2303.06726/) (2023)
- [Beyond the Independence Assumption: Finite-Sample Guarantees for Deep Q-Learning under τ-Mixing](https://mlsanity.com/paper/2605.06373/) (2026)

## Abstract (arXiv)

We prove that a single-layer neural network trained with the Q-learning algorithm converges in distribution to a random ordinary differential equation as the size of the model and the number of training steps become large. Analysis of the limit differential equation shows that it has a unique stationary solution which is the solution of the Bellman equation, thus giving the optimal control for the problem. In addition, we study the convergence of the limit differential equation to the stationary solution. As a by-product of our analysis, we obtain the limiting behavior of single-layer neural networks when trained on i.i.d. data with stochastic gradient descent under the widely-used Xavier initialization.