# Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction

arXiv: https://arxiv.org/abs/2305.02774 · HTML: https://mlsanity.com/paper/2305.02774/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 23 of 23)

- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [DuDoRNet: Learning a Dual-Domain Recurrent Network for Fast MRI Reconstruction with Deep T1 Prior](https://mlsanity.com/paper/2001.03799/) (2020)
- [PixelVAE: A Latent Variable Model for Natural Images](https://mlsanity.com/paper/1611.05013/) (2016)
- [Multi-Modal Transformer for Accelerated MR Imaging](https://mlsanity.com/paper/2106.14248/) (2021)
- [Multi-Contrast MRI Super-Resolution via a Multi-Stage Integration Network](https://mlsanity.com/paper/2105.08949/) (2021)
- [Geometric Understanding of Deep Learning](https://mlsanity.com/paper/1805.10451/) (2018)
- [Specificity-Preserving Federated Learning for MR Image Reconstruction](https://mlsanity.com/paper/2112.05752/) (2021)
- [Recurrent Variational Network: A Deep Learning Inverse Problem Solver applied to the task of Accelerated MRI Reconstruction](https://mlsanity.com/paper/2111.09639/) (2021)
- [DSFormer: A Dual-domain Self-supervised Transformer for Accelerated Multi-contrast MRI Reconstruction](https://mlsanity.com/paper/2201.10776/) (2022)
- [Over-and-Under Complete Convolutional RNN for MRI Reconstruction](https://mlsanity.com/paper/2106.08886/) (2021)
- [Multi-source Domain Adaptation via Weighted Joint Distributions Optimal Transport](https://mlsanity.com/paper/2006.12938/) (2020)
- [Unpaired Deep Learning for Accelerated MRI using Optimal Transport Driven CycleGAN](https://mlsanity.com/paper/2008.12967/) (2020)
- [Optimal Transport for Unsupervised Denoising Learning](https://mlsanity.com/paper/2108.02574/) (2021)
- [Dance Style Transfer with Cross-modal Transformer](https://mlsanity.com/paper/2208.09406/) (2022)
- [Multi-Modal MRI Reconstruction Assisted with Spatial Alignment Network](https://mlsanity.com/paper/2108.05603/) (2021)
- [AE-OT-GAN: Training GANs from data specific latent distribution](https://mlsanity.com/paper/2001.03698/) (2020)
- [Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis](https://mlsanity.com/paper/2212.01108/) (2022)
- [Dual-Domain Self-Supervised Learning for Accelerated Non-Cartesian MRI Reconstruction](https://mlsanity.com/paper/2302.09244/) (2023)
- [Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution](https://mlsanity.com/paper/2207.09156/) (2022)
- [Simultaneous Multiple-Prompt Guided Generation Using Differentiable Optimal Transport](https://mlsanity.com/paper/2204.08472/) (2022)

## Built on by (most cited first, 5 of 5)

- [A Plug-and-Play Method for Guided Multi-contrast MRI Reconstruction based on Content/Style Modeling](https://mlsanity.com/paper/2409.13477/) (2024)
- [Re-Visible Dual-Domain Self-Supervised Deep Unfolding Network for MRI Reconstruction](https://mlsanity.com/paper/2501.03737/) (2025)
- [A Comprehensive Survey on Magnetic Resonance Image Reconstruction](https://mlsanity.com/paper/2503.07097/) (2025)
- [Towards Prospective Medical Image Reconstruction via Knowledge-Informed Dynamic Optimal Transport](https://mlsanity.com/paper/2505.17644/) (2025)
- [Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models](https://mlsanity.com/paper/2509.09064/) (2025)

## Abstract (arXiv)

Multi-modal magnetic resonance imaging (MRI) plays a crucial role in comprehensive disease diagnosis in clinical medicine. However, acquiring certain modalities, such as T2-weighted images (T2WIs), is time-consuming and prone to be with motion artifacts. It negatively impacts subsequent multi-modal image analysis. To address this issue, we propose an end-to-end deep learning framework that utilizes T1-weighted images (T1WIs) as auxiliary modalities to expedite T2WIs' acquisitions. While image pre-processing is capable of mitigating misalignment, improper parameter selection leads to adverse pre-processing effects, requiring iterative experimentation and adjustment. To overcome this shortage, we employ Optimal Transport (OT) to synthesize T2WIs by aligning T1WIs and performing cross-modal synthesis, effectively mitigating spatial misalignment effects. Furthermore, we adopt an alternating iteration framework between the reconstruction task and the cross-modal synthesis task to optimize the final results. Then, we prove that the reconstructed T2WIs and the synthetic T2WIs become closer on the T2 image manifold with iterations increasing, and further illustrate that the improved reconstruction result enhances the synthesis process, whereas the enhanced synthesis result improves the reconstruction process. Finally, experimental results from FastMRI and internal datasets confirm the effectiveness of our method, demonstrating significant improvements in image reconstruction quality even at low sampling rates.