# Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning

arXiv: https://arxiv.org/abs/2609.12035 · HTML: https://mlsanity.com/paper/2609.12035/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 69 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [4M: Massively Multimodal Masked Modeling](https://mlsanity.com/paper/2312.06647/) (2023)
- [Multimodal Masked Autoencoders Learn Transferable Representations](https://mlsanity.com/paper/2205.14204/) (2022)
- [Image Captioners Are Scalable Vision Learners Too](https://mlsanity.com/paper/2306.07915/) (2023)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [USFM: A Universal Ultrasound Foundation Model Generalized to Tasks and Organs towards Label Efficient Image Analysis](https://mlsanity.com/paper/2401.00153/) (2023)
- [Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification](https://mlsanity.com/paper/2210.12843/) (2022)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model](https://mlsanity.com/paper/2502.10707/) (2025)
- [RadVLM: A Multitask Conversational Vision-Language Model for Radiology](https://mlsanity.com/paper/2502.03333/) (2025)
- [EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation](https://mlsanity.com/paper/2410.09704/) (2024)
- [USF-MAE: Ultrasound Self-Supervised Foundation Model with Masked Autoencoding](https://mlsanity.com/paper/2510.22990/) (2025)
- [Unity by Diversity: Improved Representation Learning in Multimodal VAEs](https://mlsanity.com/paper/2403.05300/) (2024)
- [GEMTrans: A General, Echocardiography-based, Multi-Level Transformer Framework for Cardiovascular Diagnosis](https://mlsanity.com/paper/2308.13217/) (2023)
- [MultiMed: Massively Multimodal and Multitask Medical Understanding](https://mlsanity.com/paper/2408.12682/) (2024)
- [Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognition](https://mlsanity.com/paper/2306.08249/) (2023)
- [A multimodal and temporal foundation model for virtual patient representations at healthcare system scale](https://mlsanity.com/paper/2604.18570/) (2026)

## Abstract (arXiv)

Cardiovascular diagnosis and treatment rest on integrating complementary modalities, such as electrocardiogram, echocardiography, and chest X-rays, each capturing distinct but complementary aspects of cardiac pathophysiology. Yet most medical foundation models remain modality-specific, combining modalities only for finetuning or post-training. This discards the cross-modal evidence clinicians naturally integrate and ignores the structure within each modality. We introduce Latent-Attention Masked Autoencoder (LAMAE), a multimodal, structure-aware masked autoencoder that jointly learns patient-level representations during self-supervised pretraining. Instead of fusing modalities post hoc, LAMAE exchanges information directly in the latent space through a shared latent-attention module operating over a study-view-entity hierarchy, enabling aggregation of variable observations and handling of missing modalities. Pretrained on over 500'000 MIMIC-IV hospital stays, LAMAE outperforms modality-specific pretraining and strong contrastive and vision-language baselines across multimodal hospital-stay tasks, such as in-hospital mortality, ICD-10 and DRG coding, and length of stay, while remaining competitive on unimodal tasks. Modeling this structure also pays off within a single modality: even without cross-modal information, the latent-attention module improves representations over modality-specific pretraining.