# Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images

arXiv: https://arxiv.org/abs/2406.00329 · HTML: https://mlsanity.com/paper/2406.00329/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 1 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Deep learning for cardiac image segmentation: A review](https://mlsanity.com/paper/1911.03723/) (2019)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentation](https://mlsanity.com/paper/2203.05573/) (2022)
- [Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences](https://mlsanity.com/paper/1806.04066/) (2018)
- [Fully Convolutional Multi-scale Residual DenseNets for Cardiac Segmentation and Automated Cardiac Diagnosis using Ensemble of Classifiers](https://mlsanity.com/paper/1801.05173/) (2018)
- [NISF: Neural Implicit Segmentation Functions](https://mlsanity.com/paper/2309.08643/) (2023)
- [Learning Interpretable Anatomical Features Through Deep Generative Models: Application to Cardiac Remodeling](https://mlsanity.com/paper/1807.06843/) (2018)
- [Learning Shape Priors for Robust Cardiac MR Segmentation from Multi-view Images](https://mlsanity.com/paper/1907.09983/) (2019)
- [MulViMotion: Shape-aware 3D Myocardial Motion Tracking from Multi-View Cardiac MRI](https://mlsanity.com/paper/2208.00034/) (2022)
- [Joint Motion Correction and Super Resolution for Cardiac Segmentation via Latent Optimisation](https://mlsanity.com/paper/2107.03887/) (2021)
- [Factorised spatial representation learning: application in semi-supervised myocardial segmentation](https://mlsanity.com/paper/1803.07031/) (2018)
- [Global k-Space Interpolation for Dynamic MRI Reconstruction using Masked Image Modeling](https://mlsanity.com/paper/2307.12672/) (2023)
- [Direct Multitype Cardiac Indices Estimation via Joint Representation and Regression Learning](https://mlsanity.com/paper/1705.09307/) (2017)
- [Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging](https://mlsanity.com/paper/2304.07675/) (2023)

## Built on by (most cited first, 9 of 9)

- [Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyond](https://mlsanity.com/paper/2504.13037/) (2025)
- [Classification, Regression and Segmentation directly from k-Space in Cardiac MRI](https://mlsanity.com/paper/2407.20108/) (2024)
- [Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registration](https://mlsanity.com/paper/2509.06467/) (2025)
- [Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG](https://mlsanity.com/paper/2506.20683/) (2025)
- [Latent Interpolation Learning Using Diffusion Models for Cardiac Volume Reconstruction](https://mlsanity.com/paper/2508.13826/) (2025)
- [Whole-body Representation Learning For Competing Preclinical Disease Risk Assessment](https://mlsanity.com/paper/2508.02307/) (2025)
- [Stochastic Siamese MAE Pretraining for Longitudinal Medical Images](https://mlsanity.com/paper/2512.23441/) (2025)
- [NISF++: Geometrically-grounded implicit representations of 3D+time cardiac function from 2D short- and long-axis MR views](https://mlsanity.com/paper/2608.00752/) (2026)
- [No Image, No Problem: End-to-End Multi-Task Cardiac Analysis from Undersampled k-Space](https://mlsanity.com/paper/2603.09945/) (2026)

## Abstract (arXiv)

Cardiac Magnetic Resonance (CMR) imaging serves as the gold-standard for evaluating cardiac morphology and function. Typically, a multi-view CMR stack, covering short-axis (SA) and 2/3/4-chamber long-axis (LA) views, is acquired for a thorough cardiac assessment. However, efficiently streamlining the complex, high-dimensional 3D+T CMR data and distilling compact, coherent representation remains a challenge. In this work, we introduce a whole-heart self-supervised learning framework that utilizes masked imaging modeling to automatically uncover the correlations between spatial and temporal patches throughout the cardiac stacks. This process facilitates the generation of meaningful and well-clustered heart representations without relying on the traditionally required, and often costly, labeled data. The learned heart representation can be directly used for various downstream tasks. Furthermore, our method demonstrates remarkable robustness, ensuring consistent representations even when certain CMR planes are missing/flawed. We train our model on 14,000 unlabeled CMR data from UK BioBank and evaluate it on 1,000 annotated data. The proposed method demonstrates superior performance to baselines in tasks that demand comprehensive 3D+T cardiac information, e.g. cardiac phenotype (ejection fraction and ventricle volume) prediction and multi-plane/multi-frame CMR segmentation, highlighting its effectiveness in extracting comprehensive cardiac features that are both anatomically and pathologically relevant.