# Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care

arXiv: https://arxiv.org/abs/2208.07406 · HTML: https://mlsanity.com/paper/2208.07406/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 37 references that are in the map.
- First posted on arXiv 15 Aug 2022.

## Builds on (most cited first, 9 of 9)

- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity](https://mlsanity.com/paper/1909.03539/) (2019)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [Inference for Batched Bandits](https://mlsanity.com/paper/2002.03217/) (2020)
- [A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients](https://mlsanity.com/paper/1605.04070/) (2016)
- [Power Constrained Bandits](https://mlsanity.com/paper/2004.06230/) (2020)
- [Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines](https://mlsanity.com/paper/2206.03944/) (2022)
- [Optimal Cost Design for Model Predictive Control](https://mlsanity.com/paper/2104.11353/) (2021)

## Built on by (most cited first, 18 of 18)

- [Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL](https://mlsanity.com/paper/2503.23157/) (2025)
- [A Deployed Online Reinforcement Learning Algorithm In An Oral Health Clinical Trial](https://mlsanity.com/paper/2409.02069/) (2024)
- [Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams](https://mlsanity.com/paper/2402.06122/) (2024)
- [Semi-parametric inference based on adaptively collected data](https://mlsanity.com/paper/2303.02534/) (2023)
- [Graph-Reward-SQL: Execution-Free Reinforcement Learning for Text-to-SQL via Graph Matching and Stepwise Reward](https://mlsanity.com/paper/2505.12380/) (2025)
- [reBandit: Random Effects based Online RL algorithm for Reducing Cannabis Use](https://mlsanity.com/paper/2402.17739/) (2024)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Online learning in bandits with predicted context](https://mlsanity.com/paper/2307.13916/) (2023)
- [Non-Stationary Latent Auto-Regressive Bandits](https://mlsanity.com/paper/2402.03110/) (2024)
- [Monitoring Fidelity of Online Reinforcement Learning Algorithms in Clinical Trials](https://mlsanity.com/paper/2402.17003/) (2024)
- [Thompson sampling for zero-inflated count outcomes with an application to the Drink Less mobile health study](https://mlsanity.com/paper/2311.14359/) (2023)
- [Oralytics Reinforcement Learning Algorithm](https://mlsanity.com/paper/2406.13127/) (2024)
- [Reinforcement Learning on Dyads to Enhance Medication Adherence](https://mlsanity.com/paper/2502.06835/) (2025)
- [OPTWIN: Drift identification with optimal sub-windows](https://mlsanity.com/paper/2305.11942/) (2023)
- [Dyadic Reinforcement Learning](https://mlsanity.com/paper/2308.07843/) (2023)
- [Prompting the Unknown: Understanding Response Uncertainty in Large Language Models](https://mlsanity.com/paper/2407.14845/) (2024)
- [Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation](https://mlsanity.com/paper/2409.10526/) (2024)
- [Can we use LLMs to bootstrap reinforcement learning? -- A case study in digital health behavior change](https://mlsanity.com/paper/2511.17630/) (2025)

## Abstract (arXiv)

Dental disease is one of the most common chronic diseases despite being largely preventable. However, professional advice on optimal oral hygiene practices is often forgotten or abandoned by patients. Therefore patients may benefit from timely and personalized encouragement to engage in oral self-care behaviors. In this paper, we develop an online reinforcement learning (RL) algorithm for use in optimizing the delivery of mobile-based prompts to encourage oral hygiene behaviors. One of the main challenges in developing such an algorithm is ensuring that the algorithm considers the impact of the current action on the effectiveness of future actions (i.e., delayed effects), especially when the algorithm has been made simple in order to run stably and autonomously in a constrained, real-world setting (i.e., highly noisy, sparse data). We address this challenge by designing a quality reward which maximizes the desired health outcome (i.e., high-quality brushing) while minimizing user burden. We also highlight a procedure for optimizing the hyperparameters of the reward by building a simulation environment test bed and evaluating candidates using the test bed. The RL algorithm discussed in this paper will be deployed in Oralytics, an oral self-care app that provides behavioral strategies to boost patient engagement in oral hygiene practices.