# The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation

arXiv: https://arxiv.org/abs/2106.04240 · HTML: https://mlsanity.com/paper/2106.04240/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs](https://mlsanity.com/paper/1706.02633/) (2017)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations](https://mlsanity.com/paper/2002.04083/) (2020)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions](https://mlsanity.com/paper/2002.03478/) (2020)
- [POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning](https://mlsanity.com/paper/2001.04032/) (2020)
- [Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs](https://mlsanity.com/paper/2006.16210/) (2020)
- [The MAGICAL Benchmark for Robust Imitation](https://mlsanity.com/paper/2011.00401/) (2020)
- [Personalized Risk Scoring for Critical Care Prognosis using Mixtures of Gaussian Processes](https://mlsanity.com/paper/1610.08853/) (2016)
- [Inverse Active Sensing: Modeling and Understanding Timely Decision-Making](https://mlsanity.com/paper/2006.14141/) (2020)
- [Simitate: A Hybrid Imitation Learning Benchmark](https://mlsanity.com/paper/1905.06002/) (2019)
- [Modelling Agent Policies with Interpretable Imitation Learning](https://mlsanity.com/paper/2006.11309/) (2020)

## Built on by (most cited first, 11 of 11)

- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Language Agents as Digital Representatives in Collective Decision-Making](https://mlsanity.com/paper/2502.09369/) (2025)
- [POETREE: Interpretable Policy Learning with Adaptive Decision Trees](https://mlsanity.com/paper/2203.08057/) (2022)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies](https://mlsanity.com/paper/2203.07338/) (2022)
- [Practical Approaches for Fair Learning with Multitype and Multivariate Sensitive Attributes](https://mlsanity.com/paper/2211.06138/) (2022)
- [Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning](https://mlsanity.com/paper/2210.05320/) (2022)
- [OCMDP: Observation-Constrained Markov Decision Process](https://mlsanity.com/paper/2411.07087/) (2024)
- [Optimising Human-AI Collaboration by Learning Convincing Explanations](https://mlsanity.com/paper/2311.07426/) (2023)
- [IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark](https://mlsanity.com/paper/2405.16069/) (2024)
- [ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects](https://mlsanity.com/paper/2111.06811/) (2021)

## Abstract (arXiv)

Understanding decision-making in clinical environments is of paramount importance if we are to bring the strengths of machine learning to ultimately improve patient outcomes. Several factors including the availability of public data, the intrinsically offline nature of the problem, and the complexity of human decision making, has meant that the mainstream development of algorithms is often geared towards optimal performance in tasks that do not necessarily translate well into the medical regime; often overlooking more niche issues commonly associated with the area. We therefore present a new benchmarking suite designed specifically for medical sequential decision making: the Medkit-Learn(ing) Environment, a publicly available Python package providing simple and easy access to high-fidelity synthetic medical data. While providing a standardised way to compare algorithms in a realistic medical setting we employ a generating process that disentangles the policy and environment dynamics to allow for a range of customisations, thus enabling systematic evaluation of algorithms' robustness against specific challenges prevalent in healthcare.