mlsanityData to 7 Oct 2026

Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity

arXiv 1909.03539 ↗First posted 8 Sep 2019cs.LG cs.AIcited by 70

Not on the map. Why?

Abstract (arXiv)

With the recent evolution of mobile health technologies, health scientists are increasingly interested in developing just-in-time adaptive interventions (JITAIs), typically delivered via notification on mobile device and designed to help the user prevent negative health outcomes and promote the adoption and maintenance of healthy behaviors. A JITAI involves a sequence of decision rules (i.e., treatment policy) that takes the user's current context as input and specifies whether and what type of an intervention should be provided at the moment. In this paper, we develop a Reinforcement Learning (RL) algorithm that continuously learns and improves the treatment policy embedded in the JITAI as the data is being collected from the user. This work is motivated by our collaboration on designing the RL algorithm in HeartSteps V2 based on data from HeartSteps V1. HeartSteps is a physical activity mobile health application. The RL algorithm developed in this paper is being used in HeartSteps V2 to decide, five times per day, whether to deliver a context-tailored activity suggestion.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 12; 2023: 15; 2024: 12; 2025: 14; 2026: 8.

70 papers.

Sort:

By area:

  1. 2021Field Study in Deploying Restless Multi-Armed Bandits: Assisting Non-Profits in Improving Maternal and Child Healthcited by 43
  2. 2021Statistical Inference with M-Estimators on Adaptively Collected Datacited by 40
  3. 2022Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelinescited by 22
  4. 2022Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Carecited by 18
  5. 2023XAIR: A Framework of Explainable AI in Augmented Realitycited by 17
  6. 2021Can Q-Learning be Improved with Advice?cited by 13
  7. 2022Counterfactual inference in sequential experimentscited by 13
  8. 2021Bandit Algorithms for Precision Medicinecited by 12
  9. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  10. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  11. 2024A Deployed Online Reinforcement Learning Algorithm In An Oral Health Clinical Trialcited by 11
  12. 2022Doubly robust nearest neighbors in factor modelscited by 9
38 more
  1. 2023Policy Optimization for Personalized Interventions in Behavioral Healthcited by 9
  2. 2024The Last JITAI? Exploring Large Language Models for Issuing Just-in-Time Adaptive Interventions: Fostering Physical Activity in a Conceptual Cardiac Rehabilitation Settingcited by 7
  3. 2025Exploring Personalized Health Support through Data-Driven, Theory-Guided LLMs: A Case Study in Sleep Healthcited by 7
  4. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  5. 2022Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventionscited by 6
  6. 2023Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resamplingcited by 6
  7. 2023Adaptive Linear Estimating Equationscited by 6
  8. 2024reBandit: Random Effects based Online RL algorithm for Reducing Cannabis Usecited by 6
  9. 2022Data-pooling Reinforcement Learning for Personalized Healthcare Interventioncited by 5
  10. 2024Time2Stop: Adaptive and Explainable Human-AI Loop for Smartphone Overuse Interventioncited by 5
  11. 2022Semi-Parametric Contextual Bandits with Graph-Laplacian Regularizationcited by 4
  12. 2023Online learning in bandits with predicted contextcited by 4
  13. 2025WatchGuardian: Enabling User-Defined Personalized Just-in-Time Intervention on Smartwatchcited by 4
  14. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  15. 2023Assessing the Impact of Context Inference Error and Partial Observability on RL Methods for Just-In-Time Adaptive Interventionscited by 3
  16. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3
  17. 2023Probabilistic Offline Policy Ranking with Approximate Bayesian Computationcited by 3
  18. 2024Temporally Multi-Scale Sparse Self-Attention for Physical Activity Data Imputationcited by 3
  19. 2024NudgeRank: Digital Algorithmic Nudging for Personalized Healthcited by 3
  20. 2021Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learningcited by 2
  21. 2023Thompson sampling for zero-inflated count outcomes with an application to the Drink Less mobile health studycited by 2
  22. 2024MiWaves Reinforcement Learning Algorithmcited by 2
  23. 2025Efficiently Solving Discounted MDPs with Predictions on Transition Matricescited by 2
  24. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  25. 2026Human-in-the-Loop Contextual Bandits for Short-Term Rental Dynamic Pricing: Structural Equivalence of Historical Warm-Up and Approval-Gated Live Learningcited by 2
  26. 2021Using Cognitive Models to Train Warm Start Reinforcement Learning Agents for Human-Computer Interactionscited by 1
  27. 2022Settling the Communication Complexity for Distributed Offline Reinforcement Learningcited by 1
  28. 2022A Bayesian Approach to Learning Bandit Structure in Markov Decision Processescited by 1
  29. 2023Decision-Focused Evaluation: Analyzing Performance of Deployed Restless Multi-Arm Banditscited by 1
  30. 2023Effect-Invariant Mechanisms for Policy Generalizationcited by 1
  31. 2023Dyadic Reinforcement Learningcited by 1
  32. 2024Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Taskscited by 1
  33. 2024The Fallacy of Minimizing Cumulative Regret in the Sequential Task Settingcited by 1
  34. 2024Oralytics Reinforcement Learning Algorithmcited by 1
  35. 2025Generator-Mediated Bandits: Thompson Sampling for GenAI-Powered Adaptive Interventionscited by 1
  36. 2025Diffusion Policies with Offline and Inverse Reinforcement Learning for Promoting Physical Activity in Older Adults Using Wearable Sensorscited by 1
  37. 2021Nonparametric Empirical Bayes Estimation and Testing for Sparse and Heteroscedastic Signalscited by 0
  38. 2021Online structural kernel selection for mobile healthcited by 0

50 most cited of 70 All 70 (JSON)