mlsanityData to 14 Oct 2026

Approximation to Deep Q-Network by Stochastic Delay Differential Equations

arXiv 2505.00382 ↗First posted 1 May 2025cs.LG math.PRcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Despite the significant breakthroughs that the Deep Q-Network (DQN) has brought to reinforcement learning, its theoretical analysis remains limited. In this paper, we construct a stochastic differential delay equation (SDDE) based on the DQN algorithm and estimate the Wasserstein-1 distance between them. We provide an upper bound for the distance and prove that the distance between the two converges to zero as the step size approaches zero. This result allows us to understand DQN's two key techniques, the experience replay and the target network, from the perspective of continuous systems. Specifically, the delay term in the equation, corresponding to the target network, contributes to the stability of the system. Our approach leverages a refined Lindeberg principle and an operator comparison to establish these results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 27 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

16 of 27 references.

References · 16

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  3. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  4. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  5. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  6. 2018Reinforcement Learning with Perturbed Rewardscited by 79
  7. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  8. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  9. 2019Neural Temporal-Difference and Q-Learning Provably Converge to Global Optimacited by 20
  10. 2023On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Explorationcited by 17
  11. 2019On Stationary-Point Hitting Time and Ergodicity of Stochastic Gradient Langevin Dynamicscited by 14
  12. 2022Understanding Deep Neural Function Approximation in Reinforcement Learning via ε-Greedy Explorationcited by 11
4 more
  1. 2021Convergence Results For Q-Learning With Experience Replaycited by 5
  2. 2021Approximation to stochastic variance reduced gradient Langevin dynamics by stochastic delay differential equationscited by 4
  3. 2020Convergence of Q-value in case of Gaussian rewardscited by 1
  4. 2024Error estimates between SGD with momentum and underdamped Langevin diffusioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Universal Approximation Theorem of Deep Q-Networks
  2. 2026Deep Q-Learning on H"older Spaces
  3. 2025DQN Performance with Epsilon Greedy Policies and Prioritized Experience Replay
  4. 2025Universal Approximation Theorem for Deep Q-Learning via FBSDE System
  5. 2025InterQ: A DQN Framework for Optimal Intermittent Control
  6. 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
  7. 2025GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
  8. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning