# ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

arXiv: https://arxiv.org/abs/2608.07418 · HTML: https://mlsanity.com/paper/2608.07418/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 107 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Placed in

- Deep reinforcement learning methods / [RL for clinical treatment and surgery](https://mlsanity.com/deeprl/rl-for-clinical-treatment-and-surgery/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents](https://mlsanity.com/paper/2405.02957/) (2024)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation](https://mlsanity.com/paper/2306.09968/) (2023)
- [AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.08755/) (2025)
- [Sequential Diagnosis with Language Models](https://mlsanity.com/paper/2506.22405/) (2025)
- [Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator](https://mlsanity.com/paper/2403.08495/) (2024)
- [LLM-as-a-Verifier: A General-Purpose Verification Framework](https://mlsanity.com/paper/2607.05391/) (2026)
- [EHRWorld: A Patient-Centric Medical World Model for Long-Horizon Clinical Trajectories](https://mlsanity.com/paper/2602.03569/) (2026)
- [Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning](https://mlsanity.com/paper/2510.04284/) (2025)
- [Evolving Interactive Diagnostic Agents in a Virtual Clinical Environment](https://mlsanity.com/paper/2510.24654/) (2025)
- [A prospective clinical feasibility study of a conversational diagnostic AI in an ambulatory primary care clinic](https://mlsanity.com/paper/2603.08448/) (2026)
- [Empowering Medical Multi-Agents with Clinical Consultation Flow for Dynamic Diagnosis](https://mlsanity.com/paper/2503.16547/) (2025)
- [Towards physician-centered oversight of conversational diagnostic AI](https://mlsanity.com/paper/2507.15743/) (2025)
- [SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment](https://mlsanity.com/paper/2605.04012/) (2026)
- [Towards Conversational Medical AI with Eyes, Ears and a Voice](https://mlsanity.com/paper/2605.09272/) (2026)
- [RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills](https://mlsanity.com/paper/2606.18203/) (2026)

## Built on by (most cited first, 3 of 3)

- [Towards Expert-level Medical AI for Real-time Video Consultations](https://mlsanity.com/paper/2608.09861/) (2026)
- [PsyCIDRA: A Dual-Agent Framework for Psychiatric Interviewing and Diagnostic Reasoning](https://mlsanity.com/paper/2610.07473/) (2026)
- [Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning](https://mlsanity.com/paper/2610.17426/) (2026)

## Abstract (arXiv)

In medical education, physicians convert academic knowledge into clinical expertise through residency: years of training across thousands of encounters, with diverse sources of feedback and progressively greater autonomy. Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters (up to 60 dialogue turns and 8 tool calls per trajectory). ResidencyRL pairs the policy agent with LLM simulators capable of complex, adversarial behaviors, training against a structured reward aligned to diagnostic accuracy, management quality, communication, documentation, and safety. On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs. 81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure. Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons. The procedural competencies transfer to unseen benchmarks: the agent outperforms the base model across all six clinical axes of the AMIE multi-visit benchmark, and shows consistent directional improvements on AgentClinic and CRAFT-MD. Our findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust, generalizable capabilities, paving the way towards clinical mastery. Prospective validation with real-world workflows remains necessary to establish clinical utility.