# Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embedding

arXiv: https://arxiv.org/abs/2506.13104 · HTML: https://mlsanity.com/paper/2506.13104/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [A Clarification of the Nuances in the Fairness Metrics Landscape](https://mlsanity.com/paper/2106.00467/) (2021)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction](https://mlsanity.com/paper/2007.10306/) (2020)
- [RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR](https://mlsanity.com/paper/2111.11665/) (2021)
- [Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation](https://mlsanity.com/paper/1912.11975/) (2019)
- [A Survey of Multimodal Information Fusion for Smart Healthcare: Mapping the Journey from Data to Wisdom](https://mlsanity.com/paper/2306.11963/) (2023)
- [Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM](https://mlsanity.com/paper/2403.08818/) (2024)
- [Mitigating Health Disparities in EHR via Deconfounder](https://mlsanity.com/paper/2210.15901/) (2022)
- [FairEHR-CLP: Towards Fairness-Aware Clinical Predictions with Contrastive Learning in Multimodal Electronic Health Records](https://mlsanity.com/paper/2402.00955/) (2024)
- [Fair Patient Model: Mitigating Bias in the Patient Representation Learned from the Electronic Health Records](https://mlsanity.com/paper/2306.03179/) (2023)
- [Debias-CLR: A Contrastive Learning Based Debiasing Method for Algorithmic Fairness in Healthcare Applications](https://mlsanity.com/paper/2411.10544/) (2024)

## Built on by (most cited first, 2 of 2)

- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction](https://mlsanity.com/paper/2607.09982/) (2026)

## Abstract (arXiv)

Electronic Health Record (EHR) data encompass diverse modalities -- text, images, and medical codes -- that are vital for clinical decision-making. To process these complex data, multimodal AI (MAI) has emerged as a powerful approach for fusing such information. However, most existing MAI models optimize for better prediction performance, potentially reinforcing biases across patient subgroups. Although bias-reduction techniques for multimodal models have been proposed, the individual strengths of each modality and their interplay in both reducing bias and optimizing performance remain underexplored. In this work, we introduce FAME (Fairness-Aware Multimodal Embeddings), a framework that explicitly weights each modality according to its fairness contribution. FAME optimizes both performance and fairness by incorporating a combined loss function. We leverage the Error Distribution Disparity Index (EDDI) to measure fairness across subgroups and propose a sign-agnostic aggregation method to balance fairness across subgroups, ensuring equitable model outcomes. We evaluate FAME with BEHRT and BioClinicalBERT, combining structured and unstructured EHR data, and demonstrate its effectiveness in terms of performance and fairness compared with other baselines across multiple EHR prediction tasks.