mlsanityData to 7 Oct 2026

Quasi-optimal Reinforcement Learning with Continuous Actions

arXiv 2301.08940 ↗First posted 21 Jan 2023stat.ML cs.LG stat.MEcited by 7

Not on the map. Why?

Abstract (arXiv)

Many real-world applications of reinforcement learning (RL) require making decisions in continuous action environments. In particular, determining the optimal dose level plays a vital role in developing medical treatment regimes. One challenge in adapting existing RL algorithms to medical applications, however, is that the popular infinite support stochastic policies, e.g., Gaussian policy, may assign riskily high dosages and harm patients seriously. Hence, it is important to induce a policy class whose support only contains near-optimal actions, and shrink the action-searching area for effectiveness and reliability. To achieve this, we develop a novel quasi-optimal learning algorithm, which can be easily optimized in off-policy settings with guaranteed convergence under general function approximations. Theoretically, we analyze the consistency, sample complexity, adaptability, and convergence of the proposed algorithm. We evaluate our algorithm with comprehensive simulated experiments and a dose suggestion real application to Ohio Type 1 diabetes dataset.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 47 of its 101 references that are in the map. First posted on arXiv 21 Jan 2023.

Builds on

47 of 101 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
35 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  4. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  7. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  9. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  10. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  11. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  12. 2014Optimizing the CVaR via Samplingcited by 98
  13. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  14. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  15. 2020Munchausen Reinforcement Learningcited by 68
  16. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  17. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  18. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
  19. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
  20. 2020Sparse and Continuous Attention Mechanismscited by 37
  21. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  22. 2019The Complexity of Finding Stationary Points with Stochastic Gradient Descentcited by 34
  23. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 29
  24. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  25. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  26. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  27. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  28. 2020Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparisoncited by 22
  29. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18
  30. 2020Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settingscited by 14
  31. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  32. 2020Causal Effect Estimation and Optimal Dose Suggestions in Mobile Healthcited by 8
  33. 2018A Parsimonious Personalized Dose Finding Model via Dimension Reductioncited by 6
  34. 2018Smoothed Action Value Functions for Learning Gaussian Policiescited by 6
  35. 2021Jump Interval-Learning for Individualized Decision Makingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 3; 2026: 1.

7 papers.

  1. 2023Bi-Level Offline Policy Optimization with Limited Explorationcited by 5
  2. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  3. 2024q-exponential family for policy optimizationcited by 2
  4. 2023AI in Pharma for Personalized Sequential Decision-Making: Methods, Applications and Opportunitiescited by 0
  5. 2025Where to Intervene: Action Selection in Deep Reinforcement Learningcited by 0
  6. 2025Symmetric Behavior Regularized Policy Optimizationcited by 0
  7. 2026Weighted Spline-Expanded Networks with Distributional Balancing for Continuous Treatment Effectscited by 0