mlsanityData to 14 Oct 2026

Making Deep Q-learning methods robust to time discretization

arXiv 1901.09732 ↗First posted 28 Jan 2019cs.LG stat.MLcited by 58

Not on the map. Why?

Cited by 28 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitivity to time discretization in near continuous-time environments as a critical factor; this covers, e.g., changing the number of frames per second, or the action frequency of the controller. Empirically, we find that Q-learning-based approaches such as Deep Q- learning (Mnih et al., 2015) and Deep Deterministic Policy Gradient (Lillicrap et al., 2015) collapse with small time steps. Formally, we prove that Q-learning does not exist in continuous time. We detail a principled way to build an off-policy RL algorithm that yields similar performances over a wide range of time discretizations, and confirm this robustness empirically.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 12; 2023: 5; 2024: 14; 2025: 9; 2026: 10.

58 papers.

Sort:

By area:

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2021Continuous-Time Model-Based Reinforcement Learningcited by 41
  4. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  5. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 26
  6. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  7. 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
  8. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 22
  9. 2022Dynamic Decision Frequency with Continuous Optionscited by 13
  10. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 13
  11. 2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 12
  12. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 12
38 more
  1. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 12
  2. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 9
  3. 2024Learning to Optimally Stop Diffusion Processes, with Financial Applicationscited by 9
  4. 2024Exploratory Optimal Stopping: A Singular Control Formulationcited by 9
  5. 2022Logarithmic regret bounds for continuous-time average-reward Markov decision processescited by 8
  6. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 8
  7. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 8
  8. 2022Linear convergence of a policy gradient method for some finite horizon continuous time control problemscited by 7
  9. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  10. 2021Continuous-Time Fitted Value Iteration for Robust Policiescited by 6
  11. 2024PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluationcited by 6
  12. 2021Towards Automatic Actor-Critic Solutions to Continuous Controlcited by 5
  13. 2022Temporal Difference Learning with Continuous Time and State in the Stochastic Settingcited by 5
  14. 2022Square-root regret bounds for continuous-time episodic Markov decision processescited by 5
  15. 2023Data-driven rules for multidimensional reflection problemscited by 5
  16. 2021ACERAC: Efficient reinforcement learning in fine time discretizationcited by 4
  17. 2022Offline Reinforcement Learning at Multiple Frequenciescited by 4
  18. 2023Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimizationcited by 4
  19. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 4
  20. 2026Policy Gradient for Continuous-Time Mean-Field Controlcited by 4
  21. 2024Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Managementcited by 3
  22. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Controlcited by 3
  23. 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
  24. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problemscited by 3
  25. 2021Continuous Homeostatic Reinforcement Learning for Self-Regulated Autonomous Agentscited by 2
  26. 2022Adaptive Asynchronous Control Using Meta-learned Neural Ordinary Differential Equationscited by 2
  27. 2022Simultaneously Updating All Persistence Values in Reinforcement Learningcited by 2
  28. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applicationscited by 2
  29. 2024An Idiosyncrasy of Time-discretization in Reinforcement Learningcited by 2
  30. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learningcited by 2
  31. 2025Continuous-Time Reinforcement Learning for Asset-Liability Managementcited by 2
  32. 2021Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learningcited by 1
  33. 2022Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-offcited by 1
  34. 2023Actor-Critic with variable time discretization via sustained actionscited by 1
  35. 2026Continuous-time q-learning for Markov regime switching system under Tsallis entropycited by 1
  36. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
  37. 2024Continuous Time Continuous Space Homeostatic Reinforcement Learning (CTCS-HRRL) : Towards Biological Self-Autonomous Agentcited by 0
  38. 2024Approximate Control for Continuous-Time POMDPscited by 0

50 most cited of 58 All 58 (JSON)