# Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Records

arXiv: https://arxiv.org/abs/2507.20993 · HTML: https://mlsanity.com/paper/2507.20993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Builds on (most cited first, 30 of 30)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes](https://mlsanity.com/paper/1704.02801/) (2017)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [On Inductive Biases for Heterogeneous Treatment Effect Estimation](https://mlsanity.com/paper/2106.03765/) (2021)
- [Causal machine learning for predicting treatment outcomes](https://mlsanity.com/paper/2410.08770/) (2024)
- [How to Make Causal Inferences Using Texts](https://mlsanity.com/paper/1802.02163/) (2018)
- [Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates](https://mlsanity.com/paper/2005.00649/) (2020)
- [Balanced Policy Evaluation and Learning](https://mlsanity.com/paper/1705.07384/) (2017)
- [Learning When-to-Treat Policies](https://mlsanity.com/paper/1905.09751/) (2019)
- [Challenges of Using Text Classifiers for Causal Inference](https://mlsanity.com/paper/1810.00956/) (2018)
- [Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects](https://mlsanity.com/paper/2111.07966/) (2021)
- [How to Leverage Multimodal EHR Data for Better Medical Predictions?](https://mlsanity.com/paper/2110.15763/) (2021)
- [Image-based Treatment Effect Heterogeneity](https://mlsanity.com/paper/2206.06417/) (2022)
- [Bounds on Representation-Induced Confounding Bias for Treatment Effect Estimation](https://mlsanity.com/paper/2311.11321/) (2023)
- [Deep EHR: Chronic Disease Prediction Using Medical Notes](https://mlsanity.com/paper/1808.04928/) (2018)
- [Conceptualizing Treatment Leakage in Text-based Causal Inference](https://mlsanity.com/paper/2205.00465/) (2022)
- [DoubleMLDeep: Estimation of Causal Effects with Multimodal Data](https://mlsanity.com/paper/2402.01785/) (2024)
- [A Comparison of Methods for Treatment Assignment with an Application to Playlist Generation](https://mlsanity.com/paper/2004.11532/) (2020)
- [Operationalizing Complex Causes: A Pragmatic View of Mediation](https://mlsanity.com/paper/2106.05074/) (2021)
- [Machine Learning Who to Nudge: Causal vs Predictive Targeting in a Field Experiment on Student Financial Aid Renewal](https://mlsanity.com/paper/2310.08672/) (2023)
- [Treatment Effect Estimation for Optimal Decision-Making](https://mlsanity.com/paper/2505.13092/) (2025)
- [Proximal Causal Inference With Text Data](https://mlsanity.com/paper/2401.06687/) (2024)
- [Optimizing Multi-Scale Representations to Detect Effect Heterogeneity Using Earth Observation and Computer Vision: Applications to Two Anti-Poverty RCTs](https://mlsanity.com/paper/2411.02134/) (2024)
- [LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding](https://mlsanity.com/paper/2507.02843/) (2025)

## Abstract (arXiv)

We study how to learn treatment policies from multimodal electronic health records (EHRs) that consist of tabular data and clinical text. These policies can help physicians make better treatment decisions and allocate healthcare resources more efficiently. Causal policy learning methods prioritize patients with the largest expected treatment benefit. Yet, existing estimators are designed for tabular covariates under causal assumptions that may be hard to justify in the multimodal setting. A pragmatic alternative is to apply causal estimators directly to multimodal representations, but this can produce biased treatment effect estimates when the representations do not preserve the relevant confounding information. As a result, predictive models of baseline risk are commonly used in practice to guide treatment decisions, although they are not designed to identify which patients benefit most from treatment. We propose AACE (Annotation-Assisted Coarsened Effects), an annotation-assisted approach to causal policy learning for multimodal EHRs. The method uses expert-provided annotations during training to support confounding adjustment, and then predicts treatment benefit from only multimodal representations at inference. We show that the proposed method achieves strong empirical performance across synthetic, semi-synthetic, and real-world EHR datasets, outperforming risk-based and representation-based causal baselines, and offering practical insights for applying causal machine learning in clinical practice.