# Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions

arXiv: https://arxiv.org/abs/2203.02605 · HTML: https://mlsanity.com/paper/2203.02605/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 335 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Finite-Time Analysis of Kernelised Contextual Bandits](https://mlsanity.com/paper/1309.6869/) (2013)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges](https://mlsanity.com/paper/1507.08025/) (2015)
- [Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation](https://mlsanity.com/paper/1807.01473/) (2018)
- [Confidence Intervals for Policy Evaluation in Adaptive Experiments](https://mlsanity.com/paper/1911.02768/) (2019)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017)
- [Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning](https://mlsanity.com/paper/1611.03531/) (2016)
- [Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2110.15332/) (2021)
- [Statistical Inference with M-Estimators on Adaptively Collected Data](https://mlsanity.com/paper/2104.14074/) (2021)
- [Residual Weighted Learning for Estimating Individualized Treatment Rules](https://mlsanity.com/paper/1508.03179/) (2015)
- [A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients](https://mlsanity.com/paper/1605.04070/) (2016)
- [Balanced Linear Contextual Bandits](https://mlsanity.com/paper/1812.06227/) (2018)
- [Recovering Bandits](https://mlsanity.com/paper/1910.14354/) (2019)
- [Assessing Time-Varying Causal Effect Moderation in Mobile Health](https://mlsanity.com/paper/1601.00237/) (2016)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Causal Decision Making and Causal Effect Estimation Are Not the Same... and Why It Matters](https://mlsanity.com/paper/2104.04103/) (2021)
- [IntelligentPooling: Practical Thompson Sampling for mHealth](https://mlsanity.com/paper/2008.01571/) (2020)
- [Action Centered Contextual Bandits](https://mlsanity.com/paper/1711.03596/) (2017)
- [Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Data](https://mlsanity.com/paper/1801.09271/) (2018)
- [Online Multi-Armed Bandits with Adaptive Inference](https://mlsanity.com/paper/2102.13202/) (2021)
- [Thompson sampling with the online bootstrap](https://mlsanity.com/paper/1410.4009/) (2014)
- [Semiparametric Contextual Bandits](https://mlsanity.com/paper/1803.04204/) (2018)
- [An Actor-Critic Contextual Bandit Algorithm for Personalized Mobile Health Interventions](https://mlsanity.com/paper/1706.09090/) (2017)
- [Structural Nested Models and G-estimation: The Partially Realized Promise](https://mlsanity.com/paper/1503.01589/) (2015)
- [Direct and indirect reinforcement learning](https://mlsanity.com/paper/1912.10600/) (2019)

## Built on by (most cited first, 6 of 6)

- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes](https://mlsanity.com/paper/2407.00364/) (2024)
- [BFTS: Thompson Sampling with Bayesian Additive Regression Trees](https://mlsanity.com/paper/2602.07767/) (2026)
- [A Meta-Learning Method for Estimation of Causal Excursion Effects to Assess Time-Varying Moderation](https://mlsanity.com/paper/2306.16297/) (2023)
- [Artificial Intelligence-based Decision Support Systems for Precision and Digital Health](https://mlsanity.com/paper/2407.16062/) (2024)
- [Conformal bandits: bringing statistical validity and reward efficiency under weak arm separability](https://mlsanity.com/paper/2512.09850/) (2025)

## Abstract (arXiv)

In recent years, reinforcement learning (RL) has acquired a prominent position in health-related sequential decision-making problems, gaining traction as a valuable tool for delivering adaptive interventions (AIs). However, in part due to a poor synergy between the methodological and the applied communities, its real-life application is still limited and its potential is still to be realized. To address this gap, our work provides the first unified technical survey on RL methods, complemented with case studies, for constructing various types of AIs in healthcare. In particular, using the common methodological umbrella of RL, we bridge two seemingly different AI domains, dynamic treatment regimes and just-in-time adaptive interventions in mobile health, highlighting similarities and differences between them and discussing the implications of using RL. Open problems and considerations for future research directions are outlined. Finally, we leverage our experience in designing case studies in both areas to showcase the significant collaborative opportunities between statistical, RL, and healthcare researchers in advancing AIs.