# MuST: Multimodal Spatiotemporal Graph-Transformer for Hospital Readmission Prediction

arXiv: https://arxiv.org/abs/2311.07608 · HTML: https://mlsanity.com/paper/2311.07608/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 11 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Targeted-BEHRT: Deep learning for observational causal inference on longitudinal electronic health records](https://mlsanity.com/paper/2202.03487/) (2022)
- [Neural networks versus Logistic regression for 30 days all-cause readmission prediction](https://mlsanity.com/paper/1812.09549/) (2018)
- [Predictive Modeling of Hospital Readmission: Challenges and Solutions](https://mlsanity.com/paper/2106.08488/) (2021)

## Built on by (most cited first, 3 of 3)

- [Prediction of 30-day hospital readmission with clinical notes and EHR information](https://mlsanity.com/paper/2503.23050/) (2025)
- [PT: A Plain Transformer is Good Hospital Readmission Predictor](https://mlsanity.com/paper/2412.12909/) (2024)
- [Mr.Dec: Daily-Scale Longitudinal Multimodal Modeling for 30-Day Readmission Prediction](https://mlsanity.com/paper/2608.16929/) (2026)

## Abstract (arXiv)

Hospital readmission prediction is considered an essential approach to decreasing readmission rates, which is a key factor in assessing the quality and efficacy of a healthcare system. Previous studies have extensively utilized three primary modalities, namely electronic health records (EHR), medical images, and clinical notes, to predict hospital readmissions. However, the majority of these studies did not integrate information from all three modalities or utilize the spatiotemporal relationships present in the dataset. This study introduces a novel model called the Multimodal Spatiotemporal Graph-Transformer (MuST) for predicting hospital readmissions. By employing Graph Convolution Networks and temporal transformers, we can effectively capture spatial and temporal dependencies in EHR and chest radiographs. We then propose a fusion transformer to combine the spatiotemporal features from the two modalities mentioned above with the features from clinical notes extracted by a pre-trained, domain-specific transformer. We assess the effectiveness of our methods using the latest publicly available dataset, MIMIC-IV. The experimental results indicate that the inclusion of multimodal features in MuST improves its performance in comparison to unimodal methods. Furthermore, our proposed pipeline outperforms the current leading methods in the prediction of hospital readmissions.