# Mesh-based 3D Motion Tracking in Cardiac MRI using Deep Learning

arXiv: https://arxiv.org/abs/2209.02004 · HTML: https://mlsanity.com/paper/2209.02004/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Self-supervised Learning of Motion Capture](https://mlsanity.com/paper/1712.01337/) (2017)
- [Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences](https://mlsanity.com/paper/1806.04066/) (2018)
- [Deep learning cardiac motion analysis for human survival prediction](https://mlsanity.com/paper/1810.03382/) (2018)
- [DARNet: Deep Active Ray Network for Building Segmentation](https://mlsanity.com/paper/1905.05889/) (2019)
- [DeepTag: An Unsupervised Deep Learning Method for Motion Tracking on Cardiac Tagging Magnetic Resonance Images](https://mlsanity.com/paper/2103.02772/) (2021)
- [Locating Objects Without Bounding Boxes](https://mlsanity.com/paper/1806.07564/) (2018)
- [Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration](https://mlsanity.com/paper/2007.02790/) (2020)
- [Biomechanics-informed Neural Networks for Myocardial Motion Tracking in MRI](https://mlsanity.com/paper/2006.04725/) (2020)
- [FOAL: Fast Online Adaptive Learning for Cardiac Motion Estimation](https://mlsanity.com/paper/2003.04492/) (2020)
- [Explainable cardiac pathology classification on cine MRI with motion characterization by semi-supervised learning of apparent flow](https://mlsanity.com/paper/1811.03433/) (2018)
- [End to End Trainable Active Contours via Differentiable Rendering](https://mlsanity.com/paper/1912.00367/) (2019)
- [Enhanced 3D Myocardial Strain Estimation from Multi-View 2D CMR Imaging](https://mlsanity.com/paper/2009.12466/) (2020)

## Built on by (most cited first, 9 of 9)

- [4D Myocardium Reconstruction with Decoupled Motion and Shape Model](https://mlsanity.com/paper/2308.14083/) (2023)
- [DeepMesh: Mesh-based Cardiac Motion Tracking using Deep Learning](https://mlsanity.com/paper/2309.14306/) (2023)
- [CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy](https://mlsanity.com/paper/2301.13098/) (2023)
- [Modeling 3D cardiac contraction and relaxation with point cloud deformation networks](https://mlsanity.com/paper/2307.10927/) (2023)
- [End-to-End 4D Heart Mesh Recovery Across Full-Stack and Sparse Cardiac MRI](https://mlsanity.com/paper/2509.12090/) (2025)
- [From Pixels to Polygons: A Survey of Deep Learning Approaches for Medical Image-to-Mesh Reconstruction](https://mlsanity.com/paper/2505.03599/) (2025)
- [CardioMorphNet: Cardiac Motion Prediction Using a Shape-Guided Bayesian Recurrent Deep Network](https://mlsanity.com/paper/2508.20734/) (2025)
- [Personalized 4D Whole-Heart Mesh Reconstruction from Cine MRI via Multi-Scale Temporal Modeling and Differentiable Contour Rendering](https://mlsanity.com/paper/2607.01952/) (2026)
- [MCM: Mamba-based Cardiac Motion Tracking using Sequential Images in MRI](https://mlsanity.com/paper/2507.17678/) (2025)

## Abstract (arXiv)

3D motion estimation from cine cardiac magnetic resonance (CMR) images is important for the assessment of cardiac function and diagnosis of cardiovascular diseases. Most of the previous methods focus on estimating pixel-/voxel-wise motion fields in the full image space, which ignore the fact that motion estimation is mainly relevant and useful within the object of interest, e.g., the heart. In this work, we model the heart as a 3D geometric mesh and propose a novel deep learning-based method that can estimate 3D motion of the heart mesh from 2D short- and long-axis CMR images. By developing a differentiable mesh-to-image rasterizer, the method is able to leverage the anatomical shape information from 2D multi-view CMR images for 3D motion estimation. The differentiability of the rasterizer enables us to train the method end-to-end. One advantage of the proposed method is that by tracking the motion of each vertex, it is able to keep the vertex correspondence of 3D meshes between time frames, which is important for quantitative assessment of the cardiac function on the mesh. We evaluate the proposed method on CMR images acquired from the UK Biobank study. Experimental results show that the proposed method quantitatively and qualitatively outperforms both conventional and learning-based cardiac motion tracking methods.