# NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting

arXiv: https://arxiv.org/abs/2609.09140 · HTML: https://mlsanity.com/paper/2609.09140/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network](https://mlsanity.com/paper/1606.00931/) (2016)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization](https://mlsanity.com/paper/2012.04456/) (2020)
- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [MOTOR: A Time-To-Event Foundation Model For Structured Medical Records](https://mlsanity.com/paper/2301.03150/) (2023)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [Effective Ways to Build and Evaluate Individual Survival Distributions](https://mlsanity.com/paper/1811.11347/) (2018)
- [Neural Fine-Gray: Monotonic neural networks for competing risks](https://mlsanity.com/paper/2305.06703/) (2023)
- [Unlocking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imaging](https://mlsanity.com/paper/2308.05764/) (2023)
- [OTIS: Learning High-Quality Time Series Features With Tiny Encoders](https://mlsanity.com/paper/2410.07299/) (2024)
- [A multimodal and temporal foundation model for virtual patient representations at healthcare system scale](https://mlsanity.com/paper/2604.18570/) (2026)
- [Time-to-Event Pretraining for 3D Medical Imaging](https://mlsanity.com/paper/2411.09361/) (2024)
- [Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG](https://mlsanity.com/paper/2506.20683/) (2025)
- [Are foundation models useful feature extractors for electroencephalography analysis?](https://mlsanity.com/paper/2502.21086/) (2025)
- [Time-Aware Attention for Enhanced Electronic Health Records Modeling](https://mlsanity.com/paper/2507.14847/) (2025)
- [Echo2ECG: Enhancing ECG Representations with Cardiac Morphology from Multi-View Echos](https://mlsanity.com/paper/2603.08505/) (2026)
- [Cross-Modal Contrastive Learning of ECG and Angiography Representations for Severe Stenosis Classification](https://mlsanity.com/paper/2606.02605/) (2026)

## Abstract (arXiv)

The digitization of healthcare has generated vast, longitudinal, and multimodal patient records over a lifetime, yet fully exploiting these data to represent and predict patient state trajectories remains a critical challenge. Current AI models often struggle to capture the complex, irregular temporal dynamics and inherent stochasticity of real-world multimodal patient data. Existing AI approaches for modeling longitudinal patient records are predominantly discriminative, limited to a few modalities, constrained by closed categorical vocabularies, treating time as a monotonic inductive bias, or they are limited in forecasting future patient states. We introduce NOAH, a time-aware, task-agnostic, generative transformer model representing and forecasting the full multimodal patient journey. NOAH features a novel bidirectional time integration and a variational latent space to capture the continuous evolution of patient states and the stochasticity of clinical trajectories. Built from over 559 million clinical events from 431,000 hospital visits of 299,000 patients across the MIMIC dataset family, NOAH natively processes medical images, time-series and numeric signals, categorical events, as well as structured and unstructured clinical records. NOAH is the first truly holistic generative model in its field, enabling autoregressive forecasting with optional time control, zero-shot classification, and counterfactual intervention simulation. It generates highly informative and predictive patient state representations that demonstrate strong performance in probing for clinical outcomes, 15 ICD chapters, and 29 comorbidities, as well as in time-to-event prediction. Seamlessly handling diverse modalities and complex temporal dynamics, NOAH provides a versatile, task-agnostic, scalable foundation for intelligent predictive systems in personalized clinical care and digital medicine.