# Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policies

arXiv: https://arxiv.org/abs/2007.12678 · HTML: https://mlsanity.com/paper/2007.12678/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2020.

## Built on by (most cited first, 24 of 24)

- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [Medical Dead-ends and Learning to Identify High-risk States and Treatments](https://mlsanity.com/paper/2110.04186/) (2021)
- [Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs](https://mlsanity.com/paper/2106.00099/) (2021)
- [Semi-Markov Offline Reinforcement Learning for Healthcare](https://mlsanity.com/paper/2203.09365/) (2022)
- [Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications](https://mlsanity.com/paper/2302.07549/) (2023)
- [User-Interactive Offline Reinforcement Learning](https://mlsanity.com/paper/2205.10629/) (2022)
- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation](https://mlsanity.com/paper/2310.17146/) (2023)
- [Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval](https://mlsanity.com/paper/2409.09430/) (2024)
- [Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agent](https://mlsanity.com/paper/2503.17553/) (2025)
- [Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/2511.20913/) (2025)
- [Pessimistic Model Selection for Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2111.14346/) (2021)
- [Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes](https://mlsanity.com/paper/2407.00364/) (2024)
- [Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care](https://mlsanity.com/paper/2306.08044/) (2023)
- [Leveraging Medical Foundation Model Features in Graph Neural Network-Based Retrieval of Breast Histopathology Images](https://mlsanity.com/paper/2405.04211/) (2024)
- [The hidden risks of temporal resampling in clinical reinforcement learning](https://mlsanity.com/paper/2602.06603/) (2026)
- [Stackelberg Batch Policy Learning](https://mlsanity.com/paper/2309.16188/) (2023)
- [Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories](https://mlsanity.com/paper/2411.05174/) (2024)
- [Learning optimal treatment strategies for intraoperative hypotension using deep reinforcement learning](https://mlsanity.com/paper/2505.21596/) (2025)
- [An Adaptive Machine Learning Triage Framework for Predicting Alzheimer's Disease Progression](https://mlsanity.com/paper/2511.06681/) (2025)
- [Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes](https://mlsanity.com/paper/2603.19440/) (2026)
- [Virtual Speech Therapist: A Clinician-in-the-Loop AI Speech Therapy Agent for Personalized and Supervised Therapy](https://mlsanity.com/paper/2605.01101/) (2026)
- [Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learning](https://mlsanity.com/paper/2606.19481/) (2026)

## Abstract (arXiv)

Standard reinforcement learning (RL) aims to find an optimal policy that identifies the best action for each state. However, in healthcare settings, many actions may be near-equivalent with respect to the reward (e.g., survival). We consider an alternative objective -- learning set-valued policies to capture near-equivalent actions that lead to similar cumulative rewards. We propose a model-free algorithm based on temporal difference learning and a near-greedy heuristic for action selection. We analyze the theoretical properties of the proposed algorithm, providing optimality guarantees and demonstrate our approach on simulated environments and a real clinical task. Empirically, the proposed algorithm exhibits good convergence properties and discovers meaningful near-equivalent actions. Our work provides theoretical, as well as practical, foundations for clinician/human-in-the-loop decision making, in which humans (e.g., clinicians, patients) can incorporate additional knowledge (e.g., side effects, patient preference) when selecting among near-equivalent actions.