# Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization

arXiv: https://arxiv.org/abs/2512.08950 · HTML: https://mlsanity.com/paper/2512.08950/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL for clinical treatment and surgery](https://mlsanity.com/deeprl/rl-for-clinical-treatment-and-surgery/)

## Builds on (most cited first, 14 of 14)

- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [Active Reinforcement Learning: Observing Rewards at a Cost](https://mlsanity.com/paper/2011.06709/) (2020)
- [Active Measure Reinforcement Learning for Observation Cost Minimization](https://mlsanity.com/paper/2005.12697/) (2020)
- [Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning](https://mlsanity.com/paper/2304.08944/) (2023)
- [Active Reinforcement Learning with Monte-Carlo Tree Search](https://mlsanity.com/paper/1803.04926/) (2018)
- [Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring](https://mlsanity.com/paper/2303.08271/) (2023)
- [Monitored Markov Decision Processes](https://mlsanity.com/paper/2402.06819/) (2024)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning](https://mlsanity.com/paper/2112.07535/) (2021)
- [Bayesian Risk-Averse Q-Learning with Streaming Observations](https://mlsanity.com/paper/2305.11300/) (2023)
- [Infinite-Horizon Linear-Quadratic-Gaussian Control with Costly Measurements](https://mlsanity.com/paper/2012.14925/) (2020)
- [Bayesian Q-learning With Imperfect Expert Demonstrations](https://mlsanity.com/paper/2210.01800/) (2022)
- [Approximate Kalman Filter Q-Learning for Continuous State-Space MDPs](https://mlsanity.com/paper/1309.6868/) (2013)
- [To Measure or Not: A Cost-Sensitive, Selective Measuring Environment for Agricultural Management Decisions with Reinforcement Learning](https://mlsanity.com/paper/2501.12823/) (2025)

## Abstract (arXiv)

Reinforcement learning in mobile health (mHealth) interventions requires balancing intervention efficacy with user burden, particularly when state measurements (for example, user surveys or feedback) are costly yet essential. The Act-Then-Measure (ATM) heuristic addresses this challenge by decoupling control and measurement actions within the Action-Contingent Noiselessly Observable Markov Decision Process (ACNO-MDP) framework. However, the standard ATM algorithm relies on a temporal-difference-inspired Q-learning method, which is prone to instability in sparse and noisy environments. In this work, we propose a Bayesian extension to ATM that replaces standard Q-learning with a Kalman filter-style Bayesian update, maintaining uncertainty-aware estimates of Q-values and enabling more stable and sample-efficient learning. We evaluate our method in both toy environments and clinically motivated testbeds. In small, tabular environments, Bayesian ATM achieves comparable or improved scalarized returns with substantially lower variance and more stable policy behavior. In contrast, in larger and more complex mHealth settings, both the standard and Bayesian ATM variants perform poorly, suggesting a mismatch between ATM's modeling assumptions and the structural challenges of real-world mHealth domains. These findings highlight the value of uncertainty-aware methods in low-data settings while underscoring the need for new RL algorithms that explicitly model causal structure, continuous states, and delayed feedback under observation cost constraints.