# Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines

arXiv: https://arxiv.org/abs/2206.03944 · HTML: https://mlsanity.com/paper/2206.03944/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 23 of 23)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Confidence Intervals for Policy Evaluation in Adaptive Experiments](https://mlsanity.com/paper/1911.02768/) (2019)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [Multi-Task Learning for Contextual Bandits](https://mlsanity.com/paper/1705.08618/) (2017)
- [A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients](https://mlsanity.com/paper/1605.04070/) (2016)
- [Power Constrained Bandits](https://mlsanity.com/paper/2004.06230/) (2020)
- [Assessing Time-Varying Causal Effect Moderation in Mobile Health](https://mlsanity.com/paper/1601.00237/) (2016)
- [IntelligentPooling: Practical Thompson Sampling for mHealth](https://mlsanity.com/paper/2008.01571/) (2020)
- [Stable discovery of interpretable subgroups via calibration in causal studies](https://mlsanity.com/paper/2008.10109/) (2020)
- [MARS-Gym: A Gym framework to model, train, and evaluate Recommender Systems for Marketplaces](https://mlsanity.com/paper/2010.07035/) (2020)
- [PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators](https://mlsanity.com/paper/2102.06961/) (2021)
- [A clustering-based reinforcement learning approach for tailored personalization of e-Health interventions](https://mlsanity.com/paper/1804.03592/) (2018)
- [Horde of Bandits using Gaussian Markov Random Fields](https://mlsanity.com/paper/1703.02626/) (2017)
- [Group-driven Reinforcement Learning for Personalized mHealth Intervention](https://mlsanity.com/paper/1708.04001/) (2017)
- [Learning to Simulate on Sparse Trajectory Data](https://mlsanity.com/paper/2103.11845/) (2021)
- [A New Method to Compare the Interpretability of Rule-based Algorithms](https://mlsanity.com/paper/2004.01570/) (2020)
- [Educational Note: Paradoxical Collider Effect in the Analysis of Non-Communicable Disease Epidemiological Data: a reproducible illustration and web application](https://mlsanity.com/paper/1809.07111/) (2018)
- [Next Waves in Veridical Network Embedding](https://mlsanity.com/paper/2007.05385/) (2020)

## Built on by (most cited first, 22 of 22)

- [Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care](https://mlsanity.com/paper/2208.07406/) (2022)
- [Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams](https://mlsanity.com/paper/2402.06122/) (2024)
- [Stable Thompson Sampling: Valid Inference via Variance Inflation](https://mlsanity.com/paper/2505.23260/) (2025)
- [reBandit: Random Effects based Online RL algorithm for Reducing Cannabis Use](https://mlsanity.com/paper/2402.17739/) (2024)
- [Efficient and robust transfer learning of optimal individualized treatment regimes with right-censored survival data](https://mlsanity.com/paper/2301.05491/) (2023)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Contextual Bandits with Budgeted Information Reveal](https://mlsanity.com/paper/2305.18511/) (2023)
- [The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning](https://mlsanity.com/paper/2306.11208/) (2023)
- [Monitoring Fidelity of Online Reinforcement Learning Algorithms in Clinical Trials](https://mlsanity.com/paper/2402.17003/) (2024)
- [Thompson sampling for zero-inflated count outcomes with an application to the Drink Less mobile health study](https://mlsanity.com/paper/2311.14359/) (2023)
- [Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Tasks](https://mlsanity.com/paper/2401.14923/) (2024)
- [Oralytics Reinforcement Learning Algorithm](https://mlsanity.com/paper/2406.13127/) (2024)
- [Reinforcement Learning on Dyads to Enhance Medication Adherence](https://mlsanity.com/paper/2502.06835/) (2025)
- [Dyadic Reinforcement Learning](https://mlsanity.com/paper/2308.07843/) (2023)
- [The Fallacy of Minimizing Cumulative Regret in the Sequential Task Setting](https://mlsanity.com/paper/2403.10946/) (2024)
- [Prompting the Unknown: Understanding Response Uncertainty in Large Language Models](https://mlsanity.com/paper/2407.14845/) (2024)
- [Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation](https://mlsanity.com/paper/2409.10526/) (2024)
- [Reproducible workflow for online AI in digital health](https://mlsanity.com/paper/2509.13499/) (2025)
- [Synthetic Data Generator for Adaptive Interventions in Global Health](https://mlsanity.com/paper/2303.01954/) (2023)
- [Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trials](https://mlsanity.com/paper/2309.14156/) (2023)
- [Can we use LLMs to bootstrap reinforcement learning? -- A case study in digital health behavior change](https://mlsanity.com/paper/2511.17630/) (2025)
- [The Evidence Ladder for Reinforcement Learning in Healthcare: From Retrospective Policies to Trusted Interventions](https://mlsanity.com/paper/2609.23374/) (2026)

## Abstract (arXiv)

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.