# An Actor-Critic Contextual Bandit Algorithm for Personalized Mobile Health Interventions

arXiv: https://arxiv.org/abs/1706.09090 · HTML: https://mlsanity.com/paper/1706.09090/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2017.

## Built on by (most cited first, 17 of 17)

- [Quantum Multi-Armed Bandits and Stochastic Linear Bandits Enjoy Logarithmic Regrets](https://mlsanity.com/paper/2205.14988/) (2022)
- [Learning Neural Contextual Bandits Through Perturbed Rewards](https://mlsanity.com/paper/2201.09910/) (2022)
- [Policy Optimization for Personalized Interventions in Behavioral Health](https://mlsanity.com/paper/2303.12206/) (2023)
- [From Personalized Medicine to Population Health: A Survey of mHealth Sensing Techniques](https://mlsanity.com/paper/2107.00948/) (2021)
- [Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions](https://mlsanity.com/paper/2203.02605/) (2022)
- [Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks](https://mlsanity.com/paper/2211.11869/) (2022)
- [Bounded Memory Adversarial Bandits with Composite Anonymous Delayed Feedback](https://mlsanity.com/paper/2204.12764/) (2022)
- [Simulated Contextual Bandits for Personalization Tasks from Recommendation Datasets](https://mlsanity.com/paper/2210.10631/) (2022)
- [BFTS: Thompson Sampling with Bayesian Additive Regression Trees](https://mlsanity.com/paper/2602.07767/) (2026)
- [Selectively Contextual Bandits](https://mlsanity.com/paper/2205.04528/) (2022)
- [Robust Tests in Online Decision-Making](https://mlsanity.com/paper/2208.09819/) (2022)
- [Inference for relative sparsity](https://mlsanity.com/paper/2306.14297/) (2023)
- [Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks](https://mlsanity.com/paper/2310.05324/) (2023)
- [Improving Reward-Conditioned Policies for Multi-Armed Bandits using Normalized Weight Functions](https://mlsanity.com/paper/2406.10795/) (2024)
- [Diabetes Lifestyle Medicine Treatment Assistance Using Reinforcement Learning](https://mlsanity.com/paper/2510.26807/) (2025)
- [Human-in-the-Loop Multi-Agent Ventilator Decision Support with Contextual Bandit Preference Learning](https://mlsanity.com/paper/2605.23320/) (2026)
- [Set Membership Estimation for Stochastic Linear Contextual Bandits](https://mlsanity.com/paper/2606.20022/) (2026)

## Abstract (arXiv)

Increasing technological sophistication and widespread use of smartphones and wearable devices provide opportunities for innovative and highly personalized health interventions. A Just-In-Time Adaptive Intervention (JITAI) uses real-time data collection and communication capabilities of modern mobile devices to deliver interventions in real-time that are adapted to the in-the-moment needs of the user. The lack of methodological guidance in constructing data-based JITAIs remains a hurdle in advancing JITAI research despite the increasing popularity of JITAIs among clinical scientists. In this article, we make a first attempt to bridge this methodological gap by formulating the task of tailoring interventions in real-time as a contextual bandit problem. Interpretability requirements in the domain of mobile health lead us to formulate the problem differently from existing formulations intended for web applications such as ad or news article placement. Under the assumption of linear reward function, we choose the reward function (the "critic") parameterization separately from a lower dimensional parameterization of stochastic policies (the "actor"). We provide an online actor-critic algorithm that guides the construction and refinement of a JITAI. Asymptotic properties of the actor-critic algorithm are developed and backed up by numerical experiments. Additional numerical experiments are conducted to test the robustness of the algorithm when idealized assumptions used in the analysis of contextual bandit algorithm are breached.