mlsanityData to 14 Oct 2026

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

arXiv 2606.04275 ↗First posted 2 Jun 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 70 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

15 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems24 Jul

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  7. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  8. 2021The Principles of Deep Learning Theorycited by 173
  9. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  10. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  11. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  12. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
2 more
  1. 2021Value Iteration in Continuous Actions, States and Timecited by 18
  2. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning
  2. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  3. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  4. 2025Action-Driven Processes for Continuous-Time Control
  5. 2025Controlling dynamics of stochastic systems with deep reinforcement learning
  6. 2026Flowing Through States: Neural ODE Regularization for Reinforcement Learning
  7. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
  8. 2026Model-Agnostic Solutions for Deep Reinforcement Learning in Non-Ergodic Contexts