# Multi-Modal Transformer for Accelerated MR Imaging

arXiv: https://arxiv.org/abs/2106.14248 · HTML: https://mlsanity.com/paper/2106.14248/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 80 references that are in the map.
- First posted on arXiv 27 Jun 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [Learning a Variational Network for Reconstruction of Accelerated MRI Data](https://mlsanity.com/paper/1704.00447/) (2017)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction](https://mlsanity.com/paper/1704.02422/) (2017)
- [Image reconstruction by domain transform manifold learning](https://mlsanity.com/paper/1704.08841/) (2017)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss](https://mlsanity.com/paper/1709.00753/) (2017)
- [Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers](https://mlsanity.com/paper/2105.08059/) (2021)
- [HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation](https://mlsanity.com/paper/1804.02967/) (2018)
- [Task Transformer Network for Joint MRI Reconstruction and Super-Resolution](https://mlsanity.com/paper/2106.06742/) (2021)
- [Deep Residual Learning for Accelerated MRI using Magnitude and Phase Networks](https://mlsanity.com/paper/1804.00432/) (2018)
- [Brain MRI Super Resolution Using 3D Deep Densely Connected Neural Networks](https://mlsanity.com/paper/1801.02728/) (2018)
- [Multi-Contrast Super-Resolution MRI Through a Progressive Network](https://mlsanity.com/paper/1908.01612/) (2019)
- [GrappaNet: Combining Parallel Imaging with Deep Learning for Multi-Coil MRI Reconstruction](https://mlsanity.com/paper/1910.12325/) (2019)
- [Multi-Contrast MRI Super-Resolution via a Multi-Stage Integration Network](https://mlsanity.com/paper/2105.08949/) (2021)
- [Deep Generalization of Structured Low-Rank Algorithms (Deep-SLR)](https://mlsanity.com/paper/1912.03433/) (2019)
- [LORAKI: Autocalibrated Recurrent Neural Networks for Autoregressive MRI Reconstruction in k-Space](https://mlsanity.com/paper/1904.09390/) (2019)
- [A Deep Information Sharing Network for Multi-contrast Compressed Sensing MRI Reconstruction](https://mlsanity.com/paper/1804.03596/) (2018)

## Built on by (most cited first, 30 of 43)

- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [ReconFormer: Accelerated MRI Reconstruction Using Recurrent Transformer](https://mlsanity.com/paper/2201.09376/) (2022)
- [Learning Federated Visual Prompt in Null Space for MRI Reconstruction](https://mlsanity.com/paper/2303.16181/) (2023)
- [Specificity-Preserving Federated Learning for MR Image Reconstruction](https://mlsanity.com/paper/2112.05752/) (2021)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusion](https://mlsanity.com/paper/2406.18950/) (2024)
- [Learning Fourier-Constrained Diffusion Bridges for MRI Reconstruction](https://mlsanity.com/paper/2308.01096/) (2023)
- [Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution](https://mlsanity.com/paper/2109.01664/) (2021)
- [PARCEL: Physics-based Unsupervised Contrastive Representation Learning for Multi-coil MR Imaging](https://mlsanity.com/paper/2202.01494/) (2022)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction](https://mlsanity.com/paper/2412.09331/) (2024)
- [A Trust-Guided Approach to MR Image Reconstruction with Side Information](https://mlsanity.com/paper/2501.03021/) (2025)
- [Deep Unfolding Convolutional Dictionary Model for Multi-Contrast MRI Super-resolution and Reconstruction](https://mlsanity.com/paper/2309.01171/) (2023)
- [Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction](https://mlsanity.com/paper/2305.02774/) (2023)
- [Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction](https://mlsanity.com/paper/2308.10910/) (2023)
- [Generative Autoregressive Transformers for Model-Agnostic Federated MRI Reconstruction](https://mlsanity.com/paper/2502.04521/) (2025)
- [Multi-branch Cascaded Swin Transformers with Attention to k-space Sampling Pattern for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2207.08412/) (2022)
- [Fast MRI Reconstruction via Edge Attention](https://mlsanity.com/paper/2304.11400/) (2023)
- [A Plug-and-Play Method for Guided Multi-contrast MRI Reconstruction based on Content/Style Modeling](https://mlsanity.com/paper/2409.13477/) (2024)
- [Accelerated Multi-Contrast MRI Reconstruction via Frequency and Spatial Mutual Learning](https://mlsanity.com/paper/2409.14113/) (2024)
- [Hierarchical Deep Feature Fusion and Ensemble Learning for Enhanced Brain Tumor MRI Classification](https://mlsanity.com/paper/2506.12363/) (2025)
- [SSCM: A Spatial-Semantic Consistent Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2509.18593/) (2025)
- [Rethinking Dual-Domain Undersampled MRI reconstruction: domain-specific design from the perspective of the receptive field](https://mlsanity.com/paper/2303.10611/) (2023)
- [DuDoUniNeXt: Dual-domain unified hybrid model for single and multi-contrast undersampled MRI reconstruction](https://mlsanity.com/paper/2403.05256/) (2024)
- [Enhance the Image: Super Resolution using Artificial Intelligence in MRI](https://mlsanity.com/paper/2406.13625/) (2024)
- [Continuous K-space Recovery Network with Image Guidance for Fast MRI Reconstruction](https://mlsanity.com/paper/2411.11282/) (2024)
- [Re-Visible Dual-Domain Self-Supervised Deep Unfolding Network for MRI Reconstruction](https://mlsanity.com/paper/2501.03737/) (2025)
- [A Comprehensive Survey on Magnetic Resonance Image Reconstruction](https://mlsanity.com/paper/2503.07097/) (2025)
- [Compound Attention and Neighbor Matching Network for Multi-contrast MRI Super-resolution](https://mlsanity.com/paper/2307.02148/) (2023)
- [The object detection method aids in image reconstruction evaluation and clinical interpretation of meniscal abnormalities](https://mlsanity.com/paper/2407.12184/) (2024)

## Abstract (arXiv)

Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we propose a multi-modal transformer (MTrans), which is capable of transferring multi-scale features from the target modality to the auxiliary modality, for accelerated MR imaging. To capture deep multi-modal information, our MTrans utilizes an improved multi-head attention mechanism, named cross attention module, which absorbs features from the auxiliary modality that contribute to the target modality. Our framework provides three appealing benefits: (i) Our MTrans use an improved transformers for multi-modal MR imaging, affording more global information compared with existing CNN-based methods. (ii) A new cross attention module is proposed to exploit the useful information in each modality at different scales. The small patch in the target modality aims to keep more fine details, the large patch in the auxiliary modality aims to obtain high-level context features from the larger region and supplement the target modality effectively. (iii) We evaluate MTrans with various accelerated multi-modal MR imaging tasks, e.g., MR image reconstruction and super-resolution, where MTrans outperforms state-of-the-art methods on fastMRI and real-world clinical datasets.