mlsanityData to 14 Oct 2026

Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning

arXiv 2110.10719 ↗First posted 20 Oct 2021stat.ME stat.MLcited by 26

Not on the map. Why?

Abstract (arXiv)

Recent advances in mobile health (mHealth) technology provide an effective way to monitor individuals' health statuses and deliver just-in-time personalized interventions. However, the practical use of mHealth technology raises unique challenges to existing methodologies on learning an optimal dynamic treatment regime. Many mHealth applications involve decision-making with large numbers of intervention options and under an infinite time horizon setting where the number of decision stages diverges to infinity. In addition, temporary medication shortages may cause optimal treatments to be unavailable, while it is unclear what alternatives can be used. To address these challenges, we propose a Proximal Temporal consistency Learning (pT-Learning) framework to estimate an optimal regime that is adaptively adjusted between deterministic and stochastic sparse policy models. The resulting minimax estimator avoids the double sampling issue in the existing algorithms. It can be further simplified and can easily incorporate off-policy data without mismatched distribution corrections. We study theoretical properties of the sparse policy and establish finite-sample bounds on the excess risk and performance error. The proposed method is provided in our proximalDTR package and is evaluated through extensive simulation studies and the OhioT1DM mHealth dataset.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 22 of its 73 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

22 of 73 references.

References · 22

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2021Convex optimizationcited by 2,515
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  6. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  7. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  8. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  9. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  10. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  11. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  12. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
10 more
  1. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  2. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  3. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 37
  4. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 30
  5. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  6. 2017On Generalized Bellman Equations and Temporal-Difference Learningcited by 8
  7. 2020Causal Effect Estimation and Optimal Dose Suggestions in Mobile Healthcited by 8
  8. 2020Latent-state models for precision medicinecited by 6
  9. 2016A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Rewardcited by 5
  10. 2015Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsizecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 9; 2024: 2; 2025: 5; 2026: 3.

26 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  3. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  4. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  5. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  6. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  7. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
  8. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  9. 2022Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventionscited by 6
  10. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  11. 2023Bi-Level Offline Policy Optimization with Limited Explorationcited by 5
  12. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
14 more
  1. 2025Reinforcement Learning for Individual Optimal Policy from Heterogeneous Datacited by 4
  2. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  3. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  4. 2023Stage-Aware Learning for Dynamic Treatmentscited by 2
  5. 2024Low-Rank Online Dynamic Assortment with Dual Contextual Informationcited by 2
  6. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  7. 2023Stackelberg Batch Policy Learningcited by 1
  8. 2024Learning Robust Treatment Rules for Censored Datacited by 1
  9. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  10. 2023AI in Pharma for Personalized Sequential Decision-Making: Methods, Applications and Opportunitiescited by 0
  11. 2025Where to Intervene: Action Selection in Deep Reinforcement Learningcited by 0
  12. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  13. 2026Counterfactually Safe Reinforcement Learningcited by 0
  14. 2026Primal--Dual Alternating Neural Learning for Timely Classification with Performance Guaranteescited by 0