# Self-supervised learning of imaging and clinical signatures using a multimodal joint-embedding predictive architecture

arXiv: https://arxiv.org/abs/2509.15470 · HTML: https://mlsanity.com/paper/2509.15470/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Builds on (most cited first, 6 of 6)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification](https://mlsanity.com/paper/2304.02836/) (2023)
- [Efficient Inference of Gaussian Process Modulated Renewal Processes with Application to Medical Event Data](https://mlsanity.com/paper/1402.4732/) (2014)
- [Body Part Regression for CT Images](https://mlsanity.com/paper/2110.09148/) (2021)
- [Unsupervised Discovery of Clinical Disease Signatures Using Probabilistic Independence](https://mlsanity.com/paper/2402.05802/) (2024)

## Built on by (most cited first, 1 of 1)

- [Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography](https://mlsanity.com/paper/2607.26196/) (2026)

## Abstract (arXiv)

The development of multimodal models for pulmonary nodule diagnosis is limited by the scarcity of labeled data and the tendency for these models to overfit on the training distribution. In this work, we leverage self-supervised learning from longitudinal and multimodal archives to address these challenges. We curate an unlabeled set of patients with CT scans and linked electronic health records from our home institution to power joint embedding predictive architecture (JEPA) pretraining. After supervised finetuning, we show that our approach outperforms an unregularized multimodal model and imaging-only model in an internal cohort (ours: 0.91, multimodal: 0.88, imaging-only: 0.73 AUC), but underperforms in an external cohort (ours: 0.72, imaging-only: 0.75 AUC). We develop a synthetic environment that characterizes the context in which JEPA may underperform. This work innovates an approach that leverages unlabeled multimodal medical archives to improve predictive models and demonstrates its advantages and limitations in pulmonary nodule diagnosis.