# Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesis

arXiv: https://arxiv.org/abs/2406.18967 · HTML: https://mlsanity.com/paper/2406.18967/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks](https://mlsanity.com/paper/1911.11897/) (2019)
- [UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translation](https://mlsanity.com/paper/2203.02557/) (2022)
- [QS-Attn: Query-Selected Attention for Contrastive Learning in I2I Translation](https://mlsanity.com/paper/2203.08483/) (2022)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Unpaired Brain MR-to-CT Synthesis using a Structure-Constrained CycleGAN](https://mlsanity.com/paper/1809.04536/) (2018)
- [Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets](https://mlsanity.com/paper/2210.05958/) (2022)
- [OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer](https://mlsanity.com/paper/2302.03453/) (2023)
- [SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2306.06289/) (2023)
- [Structure-Preserving Synthesis: MaskGAN for Unpaired MR-CT Translation](https://mlsanity.com/paper/2307.16143/) (2023)
- [CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation](https://mlsanity.com/paper/2110.06400/) (2021)
- [SA-GAN: Structure-Aware GAN for Organ-Preserving Synthetic CT Generation](https://mlsanity.com/paper/2105.07044/) (2021)
- [Unsupervised-learning-based method for chest MRI-CT transformation using structure constrained unsupervised generative attention networks](https://mlsanity.com/paper/2106.08557/) (2021)

## Built on by (most cited first, 8 of 8)

- [Anatomical feature-prioritized loss for enhanced MR to CT translation](https://mlsanity.com/paper/2410.10328/) (2024)
- [See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancement](https://mlsanity.com/paper/2512.07251/) (2025)
- [Cyclic Self-Supervised Diffusion for Ultra Low-field to High-field MRI Synthesis](https://mlsanity.com/paper/2510.13735/) (2025)
- [Pretext Task Adversarial Learning for Unpaired Low-field to Ultra High-field MRI Synthesis](https://mlsanity.com/paper/2503.05339/) (2025)
- [LatXGen: Towards Radiation-Free and Accurate Quantitative Analysis of Sagittal Spinal Alignment Via Cross-Modal Radiographic View Synthesis](https://mlsanity.com/paper/2509.24165/) (2025)
- [Flip Distribution Alignment VAE for Multi-Phase MRI Synthesis](https://mlsanity.com/paper/2510.02970/) (2025)
- [Radiology Report Generation with Layer-Wise Anatomical Attention](https://mlsanity.com/paper/2512.16841/) (2025)
- [WING: A Window-Prior-Based Generative Network with Gated Inception for Cross-Modality CT Synthesis](https://mlsanity.com/paper/2607.06234/) (2026)

## Abstract (arXiv)

Unpaired medical image synthesis aims to provide complementary information for an accurate clinical diagnostics, and address challenges in obtaining aligned multi-modal medical scans. Transformer-based models excel in imaging translation tasks thanks to their ability to capture long-range dependencies. Although effective in supervised training settings, their performance falters in unpaired image synthesis, particularly in synthesizing structural details. This paper empirically demonstrates that, lacking strong inductive biases, Transformer can converge to non-optimal solutions in the absence of paired data. To address this, we introduce UNet Structured Transformer (UNest), a novel architecture incorporating structural inductive biases for unpaired medical image synthesis. We leverage the foundational Segment-Anything Model to precisely extract the foreground structure and perform structural attention within the main anatomy. This guides the model to learn key anatomical regions, thus improving structural synthesis under the lack of supervision in unpaired training. Evaluated on two public datasets, spanning three modalities, i.e., MR, CT, and PET, UNest improves recent methods by up to 19.30% across six medical image synthesis tasks. Our code is released at https://github.com/HieuPhan33/MICCAI2024-UNest.