# Deformation-aware GAN for Medical Image Synthesis with Substantially Misaligned Pairs

arXiv: https://arxiv.org/abs/2408.09432 · HTML: https://mlsanity.com/paper/2408.09432/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1809.05852/) (2018)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Cycle-Consistent Counterfactuals by Latent Transformations](https://mlsanity.com/paper/2203.15064/) (2022)
- [Guided Image-to-Image Translation with Bi-Directional Feature Transformation](https://mlsanity.com/paper/1910.11328/) (2019)
- [Deep learning-based synthetic-CT generation in radiotherapy and PET: a review](https://mlsanity.com/paper/2102.02734/) (2021)
- [Modulated Contrast for Versatile Image Synthesis](https://mlsanity.com/paper/2203.09333/) (2022)
- [Inverse Consistency by Construction for Multistep Deep Registration](https://mlsanity.com/paper/2305.00087/) (2023)
- [Discriminative Region Proposal Adversarial Networks for High-Quality Image-to-Image Translation](https://mlsanity.com/paper/1711.09554/) (2017)
- [Maximum Spatial Perturbation Consistency for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2203.12707/) (2022)
- [One-to-one Mapping for Unpaired Image-to-image Translation](https://mlsanity.com/paper/1909.04110/) (2019)

## Built on by (most cited first, 2 of 2)

- [TraceTrans: Translation and Spatial Tracing for Surgical Prediction](https://mlsanity.com/paper/2510.22379/) (2025)
- [When Misalignment Becomes Supervision: Structured Label Noise in Supervised Synthetic CT Generation](https://mlsanity.com/paper/2609.29387/) (2026)

## Abstract (arXiv)

Medical image synthesis generates additional imaging modalities that are costly, invasive or harmful to acquire, which helps to facilitate the clinical workflow. When training pairs are substantially misaligned (e.g., lung MRI-CT pairs with respiratory motion), accurate image synthesis remains a critical challenge. Recent works explored the directional registration module to adjust misalignment in generative adversarial networks (GANs); however, substantial misalignment will lead to 1) suboptimal data mapping caused by correspondence ambiguity, and 2) degraded image fidelity caused by morphology influence on discriminators. To address the challenges, we propose a novel Deformation-aware GAN (DA-GAN) to dynamically correct the misalignment during the image synthesis based on multi-objective inverse consistency. Specifically, in the generative process, three levels of inverse consistency cohesively optimise symmetric registration and image generation for improved correspondence. In the adversarial process, to further improve image fidelity under misalignment, we design deformation-aware discriminators to disentangle the mismatched spatial morphology from the judgement of image fidelity. Experimental results show that DA-GAN achieved superior performance on a public dataset with simulated misalignments and a real-world lung MRI-CT dataset with respiratory motion misalignment. The results indicate the potential for a wide range of medical image synthesis tasks such as radiotherapy planning.