mlsanityData to 7 Oct 2026

Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health

arXiv 1912.13088 ↗First posted 30 Dec 2019cs.LG math.ST stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

Due to the recent advancements in wearables and sensing technology, health scientists are increasingly developing mobile health (mHealth) interventions. In mHealth interventions, mobile devices are used to deliver treatment to individuals as they go about their daily lives. These treatments are generally designed to impact a near time, proximal outcome such as stress or physical activity. The mHealth intervention policies, often called just-in-time adaptive interventions, are decision rules that map an individual's current state (e.g., individual's past behaviors as well as current observations of time, location, social activity, stress and urges to smoke) to a particular treatment at each of many time points. The vast majority of current mHealth interventions deploy expert-derived policies. In this paper, we provide an approach for conducting inference about the performance of one or more such policies using historical data collected under a possibly different policy. Our measure of performance is the average of proximal outcomes over a long time period should the particular mHealth policy be followed. We provide an estimator as well as confidence intervals. This work is motivated by HeartSteps, an mHealth physical activity intervention.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 6; 2022: 11; 2023: 10; 2024: 7; 2025: 6; 2026: 1.

42 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2021Universal Off-Policy Evaluationcited by 43
  3. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  4. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  5. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  6. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  7. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  8. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  9. 2022Conformal Off-policy Predictioncited by 14
  10. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  11. 2020Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learningcited by 8
  12. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
30 more
  1. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  2. 2023Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharingcited by 7
  3. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  4. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  5. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 7
  6. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  7. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  8. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  9. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  10. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  11. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  12. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  13. 2022Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Makingcited by 4
  14. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  15. 2023Value-aware Importance Weighting for Off-policy Reinforcement Learningcited by 4
  16. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  17. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  18. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  19. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  20. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  21. 2022Importance Sampling Placement in Off-Policy Temporal-Difference Methodscited by 1
  22. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  23. 2022Statistical Learning for Individualized Asset Allocationcited by 0
  24. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  25. 2023Asymptotic Inference for Multi-Stage Stationary Treatment Policy with Variable Selectioncited by 0
  26. 2024Short-Long Policy Evaluation with Novel Actionscited by 0
  27. 2025DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effectscited by 0
  28. 2025An Orthogonal Learner for Individualized Outcomes in Markov Decision Processescited by 0
  29. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  30. 2026Distributional Off-Policy Evaluation with Deep Quantile Process Regressioncited by 0