# PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer

arXiv: https://arxiv.org/abs/2105.13993 · HTML: https://mlsanity.com/paper/2105.13993/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks](https://mlsanity.com/paper/1506.05751/) (2015)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.04357/) (2016)
- [Infant FreeSurfer: An automated segmentation and surface extraction pipeline for T1-weighted neuroimaging data of infants 0-2 years](https://mlsanity.com/paper/2001.03091/) (2020)

## Built on by (most cited first, 11 of 11)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Swin Transformer for Fast MRI](https://mlsanity.com/paper/2201.03230/) (2022)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation](https://mlsanity.com/paper/2204.13738/) (2022)
- [I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling](https://mlsanity.com/paper/2405.14022/) (2024)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis](https://mlsanity.com/paper/2212.01108/) (2022)
- [McCaD: Multi-Contrast MRI Conditioned, Adaptive Adversarial Diffusion Model for High-Fidelity MRI Synthesis](https://mlsanity.com/paper/2409.00585/) (2024)
- [Pyramid Hierarchical Masked Diffusion Model for Imaging Synthesis](https://mlsanity.com/paper/2507.16579/) (2025)
- [Flip Distribution Alignment VAE for Multi-Phase MRI Synthesis](https://mlsanity.com/paper/2510.02970/) (2025)

## Abstract (arXiv)

Magnetic resonance imaging (MRI) noninvasively provides critical information about how human brain structures develop across stages of life. Developmental scientists are particularly interested in the first few years of neurodevelopment. Despite the success of MRI collection and analysis for adults, it is a challenge for researchers to collect high-quality multimodal MRIs from developing infants mainly because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still, and a lack of analysis approaches. These challenges often lead to a significant reduction of usable data. To address this issue, researchers have explored various solutions to replace corrupted scans through synthesizing realistic MRIs. Among them, the convolution neural network (CNN) based generative adversarial network has demonstrated promising results and achieves state-of-the-art performance. However, adversarial training is unstable and may need careful tuning of regularization terms to stabilize the training. In this study, we introduced a novel MRI synthesis framework - Pyramid Transformer Net (PTNet). PTNet consists of transformer layers, skip-connections, and multi-scale pyramid representation. Compared with the most widely used CNN-based conditional GAN models (namely pix2pix and pix2pixHD), our model PTNet shows superior performance in terms of synthesis accuracy and model size. Notably, PTNet does not require any type of adversarial training and can be easily trained using the simple mean squared error loss.