# Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis

arXiv: https://arxiv.org/abs/2212.01108 · HTML: https://mlsanity.com/paper/2212.01108/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 62 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images](https://mlsanity.com/paper/2004.14133/) (2020)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration](https://mlsanity.com/paper/2104.06468/) (2021)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation](https://mlsanity.com/paper/2204.13738/) (2022)
- [Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection](https://mlsanity.com/paper/1802.07846/) (2018)
- [Dual-Sampling Attention Network for Diagnosis of COVID-19 from Community Acquired Pneumonia](https://mlsanity.com/paper/2005.02690/) (2020)
- [Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis](https://mlsanity.com/paper/2008.08698/) (2020)
- [Multi-Instance Multi-Scale CNN for Medical Image Classification](https://mlsanity.com/paper/1907.02413/) (2019)
- [TarGAN: Target-Aware Generative Adversarial Networks for Multi-modality Medical Image Translation](https://mlsanity.com/paper/2105.08993/) (2021)
- [PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer](https://mlsanity.com/paper/2105.13993/) (2021)
- [POCFormer: A Lightweight Transformer Architecture for Detection of COVID-19 Using Point of Care Ultrasound](https://mlsanity.com/paper/2105.09913/) (2021)

## Built on by (most cited first, 13 of 13)

- [I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling](https://mlsanity.com/paper/2405.14022/) (2024)
- [Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction](https://mlsanity.com/paper/2305.02774/) (2023)
- [TriDo-Former: A Triple-Domain Transformer for Direct PET Reconstruction from Low-Dose Sinograms](https://mlsanity.com/paper/2308.05365/) (2023)
- [CAVM: Conditional Autoregressive Vision Model for Contrast-Enhanced Brain Tumor MRI Synthesis](https://mlsanity.com/paper/2406.16074/) (2024)
- [Image-to-Image Translation Framework Embedded with Rotation Symmetry Priors](https://mlsanity.com/paper/2604.12805/) (2026)
- [Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning](https://mlsanity.com/paper/2406.08634/) (2024)
- [Fully Guided Neural Schr"odinger bridge for Brain MR image synthesis](https://mlsanity.com/paper/2501.14171/) (2025)
- [U2AD: Uncertainty-based Unsupervised Anomaly Detection Framework for Detecting T2 Hyperintensity in MRI Spinal Cord](https://mlsanity.com/paper/2503.13400/) (2025)
- [Bringing together invertible UNets with invertible attention modules for memory-efficient diffusion models](https://mlsanity.com/paper/2504.10883/) (2025)
- [Flip Distribution Alignment VAE for Multi-Phase MRI Synthesis](https://mlsanity.com/paper/2510.02970/) (2025)
- [Brain MR Image Synthesis with 3D Multi-Contrast Self-Attention GAN](https://mlsanity.com/paper/2604.00070/) (2026)
- [An Attention-based Model for Robust Forecasting with Missing Modality](https://mlsanity.com/paper/2606.13970/) (2026)
- [S3-Diff: Structural Semantic Synergy Diffusion Model for High Fidelity Super Resolution of Pathological Images](https://mlsanity.com/paper/2608.03540/) (2026)

## Abstract (arXiv)

Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. However, it is often challenging to obtain sufficient paired data for supervised training. In reality, we often have a small number of paired data while a large number of unpaired data. To take advantage of both paired and unpaired data, in this paper, we propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis. Specifically, an Edge-preserving Masked AutoEncoder (Edge-MAE) is first pre-trained in a self-supervised manner to simultaneously perform 1) image imputation for randomly masked patches in each image and 2) whole edge map estimation, which effectively learns both contextual and structural information. Besides, a novel patch-wise loss is proposed to enhance the performance of Edge-MAE by treating different masked patches differently according to the difficulties of their respective imputations. Based on this proposed pre-training, in the subsequent fine-tuning stage, a Dual-scale Selective Fusion (DSF) module is designed (in our MT-Net) to synthesize missing-modality images by integrating multi-scale features extracted from the encoder of the pre-trained Edge-MAE. Further, this pre-trained encoder is also employed to extract high-level features from the synthesized image and corresponding ground-truth image, which are required to be similar (consistent) in the training. Experimental results show that our MT-Net achieves comparable performance to the competing methods even using 70% of all available paired data. Our code will be publicly available at https://github.com/lyhkevin/MT-Net.