mlsanityData to 14 Oct 2026

Evaluating Reinforcement Learning Algorithms in Observational Health Settings

arXiv 1805.12298 ↗First posted 31 May 2018cs.LG stat.MLcited by 60

Not on the map. Why?

Cited by 29 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effects. Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation, sepsis management and treatment of schizophrenia. However, before implementing treatment policies learned by black-box algorithms in high-stakes clinical decision problems, special care must be taken in the evaluation of these policies. In this document, our goal is to expose some of the subtleties associated with evaluating RL algorithms in healthcare. We aim to provide a conceptual starting point for clinical and computational researchers to ask the right questions when designing and evaluating algorithms for new ways of treating patients. In the following, we describe how choices about how to summarize a history, variance of statistical estimators, and confounders in more ad-hoc measures can result in unreliable, even misleading estimates of the quality of a treatment policy. We also provide suggestions for mitigating these effects---for while there is much promise for mining observational health data to uncover better treatment policies, evaluation must be performed thoughtfully.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 4; 2021: 15; 2022: 9; 2023: 9; 2024: 10; 2025: 5; 2026: 7.

60 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  4. 2021Towards Causal Representation Learningcited by 209
  5. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  6. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  7. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  8. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
  9. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  10. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  11. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  12. 2021Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatmentcited by 16
38 more
  1. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  2. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  3. 2022Federated Offline Reinforcement Learningcited by 13
  4. 2019Inverse Reinforcement Learning in Contextual MDPscited by 12
  5. 2021Learning-to-defer for sequential medical decision-making under uncertaintycited by 12
  6. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  7. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  8. 2024DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regimecited by 8
  9. 2021Challenges for Reinforcement Learning in Healthcarecited by 7
  10. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  11. 2023Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluationcited by 7
  12. 2021Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluationcited by 5
  13. 2022Deconfounding Actor-Critic Network with Policy Adaptation for Dynamic Treatment Regimescited by 5
  14. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexaminationcited by 5
  15. 2023Get Back Here: Robust Imitation by Return-to-Distribution Planningcited by 4
  16. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  17. 2021Reducing Conservativeness Oriented Offline Reinforcement Learningcited by 3
  18. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learningcited by 3
  19. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  20. 2023Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Carecited by 2
  21. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
  22. 2025Outcome-Aware Spectral Feature Learning for Instrumental Variable Regressioncited by 2
  23. 2026EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twinscited by 2
  24. 2021Case-based off-policy policy evaluation using prototype learningcited by 1
  25. 2022Deep Normed Embeddings for Patient Representationcited by 1
  26. 2023Offline Reinforcement Learning with Additional Covering Distributionscited by 1
  27. 2024Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learningcited by 1
  28. 2024Methodology for Interpretable Reinforcement Learning for Optimizing Mechanical Ventilationcited by 1
  29. 2021An Analysis of Reinforcement Learning for Malaria Controlcited by 0
  30. 2021Continuous Treatment Recommendation with Deep Survival Dose Response Functioncited by 0
  31. 2021Compositional Q-learning for electrolyte repletion with imbalanced patient sub-populationscited by 0
  32. 2022Reinforcement Learning For Survival, A Clinically Motivated Method For Critically Ill Patientscited by 0
  33. 2022Towards Robust Off-Policy Evaluation via Human Inputscited by 0
  34. 2022Wall Street Tree Search: Risk-Aware Planning for Offline Reinforcement Learningcited by 0
  35. 2023Deep Attention Q-Network for Personalized Treatment Recommendationcited by 0
  36. 2024Optimizing Warfarin Dosing Using Contextual Bandit: An Offline Policy Learning and Evaluation Methodcited by 0
  37. 2024Curricula for Learning Robust Policies with Factored State Representations in Changing Environmentscited by 0
  38. 2024How Should We Represent History in Interpretable Models of Clinical Policies?cited by 0

50 most cited of 60 All 60 (JSON)