mlsanityData to 7 Oct 2026

Policy Optimization for Continuous Reinforcement Learning

arXiv 2305.18901 ↗First posted 30 May 2023cs.LG math.OCcited by 30

Not on the map. Why?

Abstract (arXiv)

We study reinforcement learning (RL) in the setting of continuous time and space, for an infinite horizon with a discounted objective and the underlying dynamics driven by a stochastic differential equation. Built upon recent advances in the continuous approach to RL, we develop a notion of occupation time (specifically for a discounted objective), and show how it can be effectively used to derive performance-difference and local-approximation formulas. We further extend these results to illustrate their applications in the PG (policy gradient) and TRPO/PPO (trust region policy optimization/ proximal policy optimization) methods, which have been familiar and powerful tools in the discrete RL setting but under-developed in continuous RL. Through numerical experiments, we demonstrate the effectiveness and advantages of our approach.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 31 of its 66 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

31 of 66 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  10. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  11. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  12. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
19 more
  1. 2018Deep Hedgingcited by 106
  2. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  3. 2022q-Learning in Continuous Timecited by 68
  4. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  5. 2019Making Deep Q-learning methods robust to time discretizationcited by 51
  6. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 42
  7. 2020Queueing Network Controls via Deep Reinforcement Learningcited by 40
  8. 2020Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEscited by 32
  9. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 31
  10. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 29
  11. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  12. 2021Exploratory HJB equations and their convergencecited by 28
  13. 2022Deep Inventory Managementcited by 22
  14. 2021Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controlscited by 21
  15. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 20
  16. 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 18
  17. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
  18. 2021Nonconvex Optimization via MM Algorithms: Convergence Theorycited by 5
  19. 2021Refined Policy Improvement Bounds for MDPscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 9; 2026: 10.

30 papers.

Sort:

By area:

  1. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  2. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorialcited by 60
  3. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  4. 2024Contractive Diffusion Probabilistic Modelscited by 29
  5. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  6. 2024Regret of exploratory policy improvement and q-learningcited by 10
  7. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 9
  8. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 9
  9. 2025Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Modelscited by 9
  10. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  11. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 7
  12. 2023Learning an Inventory Control Policy with General Inventory Arrival Dynamicscited by 5
18 more
  1. 2024Neural Coordination and Capacity Control for Inventory Managementcited by 4
  2. 2024Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Managementcited by 3
  3. 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
  4. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  5. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 3
  6. 2024A Differential and Pointwise Control Approach to Reinforcement Learningcited by 1
  7. 2025Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergencecited by 1
  8. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  9. 2025Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selectioncited by 0
  10. 2025Interpretable Hybrid Deep Q-Learning Framework for IoT-Based Food Spoilage Prediction with Synthetic Data Generation and Hardware Validationcited by 0
  11. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0
  12. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flowscited by 0
  13. 2026Privacy Preserving Reinforcement Learning with One-Sided Feedbackcited by 0
  14. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0
  15. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 0
  16. 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learningcited by 0
  17. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  18. 2026Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricingcited by 0