mlsanityData to 14 Oct 2026

Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines

arXiv 2206.03944 ↗First posted 8 Jun 2022cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

23 of 51 references.

References · 23

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2016OpenAI Gymcited by 1,736
  3. 2017A Tutorial on Thompson Samplingcited by 433
  4. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  5. 2019Confidence Intervals for Policy Evaluation in Adaptive Experimentscited by 95
  6. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 94
  7. 2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
  8. 2019Veridical Data Sciencecited by 66
  9. 2017Multi-Task Learning for Contextual Banditscited by 47
  10. 2016A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patientscited by 38
  11. 2020Power Constrained Banditscited by 30
  12. 2016Assessing Time-Varying Causal Effect Moderation in Mobile Healthcited by 27
11 more
  1. 2020IntelligentPooling: Practical Thompson Sampling for mHealthcited by 24
  2. 2020Stable discovery of interpretable subgroups via calibration in causal studiescited by 18
  3. 2020MARS-Gym: A Gym framework to model, train, and evaluate Recommender Systems for Marketplacescited by 13
  4. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulatorscited by 10
  5. 2018A clustering-based reinforcement learning approach for tailored personalization of e-Health interventionscited by 8
  6. 2017Horde of Bandits using Gaussian Markov Random Fieldscited by 6
  7. 2017Group-driven Reinforcement Learning for Personalized mHealth Interventioncited by 6
  8. 2021Learning to Simulate on Sparse Trajectory Datacited by 6
  9. 2020A New Method to Compare the Interpretability of Rule-based Algorithmscited by 5
  10. 2018Educational Note: Paradoxical Collider Effect in the Analysis of Non-Communicable Disease Epidemiological Data: a reproducible illustration and web applicationcited by 4
  11. 2020Next Waves in Veridical Network Embeddingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 8; 2025: 4; 2026: 2.

22 papers.

Sort:

By area:

  1. 2022Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Carecited by 18
  2. 2024Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streamscited by 9
  3. 2025Stable Thompson Sampling: Valid Inference via Variance Inflationcited by 8
  4. 2024reBandit: Random Effects based Online RL algorithm for Reducing Cannabis Usecited by 6
  5. 2023Efficient and robust transfer learning of optimal individualized treatment regimes with right-censored survival datacited by 5
  6. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  7. 2023Contextual Bandits with Budgeted Information Revealcited by 3
  8. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3
  9. 2024Monitoring Fidelity of Online Reinforcement Learning Algorithms in Clinical Trialscited by 3
  10. 2023Thompson sampling for zero-inflated count outcomes with an application to the Drink Less mobile health studycited by 2
  11. 2024Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Taskscited by 2
  12. 2024Oralytics Reinforcement Learning Algorithmcited by 2
10 more
  1. 2025Reinforcement Learning on Dyads to Enhance Medication Adherencecited by 2
  2. 2023Dyadic Reinforcement Learningcited by 1
  3. 2024The Fallacy of Minimizing Cumulative Regret in the Sequential Task Settingcited by 1
  4. 2024Prompting the Unknown: Understanding Response Uncertainty in Large Language Modelscited by 1
  5. 2024Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementationcited by 1
  6. 2025Reproducible workflow for online AI in digital healthcited by 1
  7. 2023Synthetic Data Generator for Adaptive Interventions in Global Healthcited by 0
  8. 2023Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trialscited by 0
  9. 2025Can we use LLMs to bootstrap reinforcement learning? -- A case study in digital health behavior changecited by 0
  10. 2026The Evidence Ladder for Reinforcement Learning in Healthcare: From Retrospective Policies to Trusted Interventionscited by 0