# Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning

arXiv: https://arxiv.org/abs/1611.03531 · HTML: https://mlsanity.com/paper/1611.03531/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2016.

## Built on by (most cited first, 30 of 65)

- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process](https://mlsanity.com/paper/2202.10589/) (2022)
- [A Statistical Analysis of Polyak-Ruppert Averaged Q-learning](https://mlsanity.com/paper/2112.14582/) (2021)
- [Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2108.03706/) (2021)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability](https://mlsanity.com/paper/2110.12343/) (2021)
- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Projected State-action Balancing Weights for Offline Reinforcement Learning](https://mlsanity.com/paper/2109.04640/) (2021)
- [Conformal Off-policy Prediction](https://mlsanity.com/paper/2206.06711/) (2022)
- [Federated Offline Reinforcement Learning](https://mlsanity.com/paper/2206.05581/) (2022)
- [Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2210.11846/) (2022)
- [Testing Stationarity and Change Point Detection in Reinforcement Learning](https://mlsanity.com/paper/2203.01707/) (2022)
- [Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons](https://mlsanity.com/paper/2202.13163/) (2022)
- [Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning](https://mlsanity.com/paper/2006.03886/) (2020)
- [Policy Evaluation for Temporal and/or Spatial Dependent Experiments](https://mlsanity.com/paper/2202.10887/) (2022)
- [Reinforcement Learning with Heterogeneous Data: Estimation and Inference](https://mlsanity.com/paper/2202.00088/) (2022)
- [Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimization](https://mlsanity.com/paper/2301.02220/) (2023)
- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [Sequential Knockoffs for Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2303.14281/) (2023)
- [Off-policy Evaluation in Doubly Inhomogeneous Environments](https://mlsanity.com/paper/2306.08719/) (2023)
- [Robust Offline Reinforcement learning with Heavy-Tailed Rewards](https://mlsanity.com/paper/2310.18715/) (2023)
- [Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approach](https://mlsanity.com/paper/2112.04571/) (2021)
- [Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions](https://mlsanity.com/paper/2203.02605/) (2022)
- [Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Framework](https://mlsanity.com/paper/2309.13278/) (2023)
- [Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments](https://mlsanity.com/paper/2403.17285/) (2024)
- [Combining Experimental and Historical Data for Policy Evaluation](https://mlsanity.com/paper/2406.00317/) (2024)
- [Quantile Off-Policy Evaluation via Deep Conditional Generative Learning](https://mlsanity.com/paper/2212.14466/) (2022)
- [Revisiting Bellman Errors for Offline Model Selection](https://mlsanity.com/paper/2302.00141/) (2023)
- [Bi-Level Offline Policy Optimization with Limited Exploration](https://mlsanity.com/paper/2310.06268/) (2023)

## Abstract (arXiv)

The vision for precision medicine is to use individual patient characteristics to inform a personalized treatment plan that leads to the best healthcare possible for each patient. Mobile technologies have an important role to play in this vision as they offer a means to monitor a patient's health status in real-time and subsequently to deliver interventions if, when, and in the dose that they are needed. Dynamic treatment regimes formalize individualized treatment plans as sequences of decision rules, one per stage of clinical intervention, that map current patient information to a recommended treatment. However, existing methods for estimating optimal dynamic treatment regimes are designed for a small number of fixed decision points occurring on a coarse time-scale. We propose a new reinforcement learning method for estimating an optimal treatment regime that is applicable to data collected using mobile technologies in an outpatient setting. The proposed method accommodates an indefinite time horizon and minute-by-minute decision making that are common in mobile health applications. We show the proposed estimators are consistent and asymptotically normal under mild conditions. The proposed methods are applied to estimate an optimal dynamic treatment regime for controlling blood glucose levels in patients with type 1 diabetes.