# Deformation equivariant cross-modality image synthesis with paired non-aligned training data

arXiv: https://arxiv.org/abs/2208.12491 · HTML: https://mlsanity.com/paper/2208.12491/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 75 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration](https://mlsanity.com/paper/1809.06130/) (2018)
- [Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning](https://mlsanity.com/paper/2112.04489/) (2021)
- [Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration](https://mlsanity.com/paper/1805.04605/) (2018)
- [Deep Learning in Medical Image Registration: A Review](https://mlsanity.com/paper/1912.12318/) (2019)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation](https://mlsanity.com/paper/2003.08073/) (2020)
- [Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2003.00273/) (2020)
- [CoMIR: Contrastive Multimodal Image Representation for Registration](https://mlsanity.com/paper/2006.06325/) (2020)
- [Cross-modality image synthesis from unpaired data using CycleGAN: Effects of gradient consistency loss and training data size](https://mlsanity.com/paper/1803.06629/) (2018)
- [PhaseStain: Digital staining of label-free quantitative phase microscopy images using deep learning](https://mlsanity.com/paper/1807.07701/) (2018)
- [Deep CT to MR Synthesis using Paired and Unpaired Data](https://mlsanity.com/paper/1805.10790/) (2018)
- [Deep learning-based synthetic-CT generation in radiotherapy and PET: a review](https://mlsanity.com/paper/2102.02734/) (2021)
- [Evaluating the Impact of Intensity Normalization on MR Image Synthesis](https://mlsanity.com/paper/1812.04652/) (2018)
- [GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis](https://mlsanity.com/paper/1901.04059/) (2019)
- [Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation](https://mlsanity.com/paper/2204.13656/) (2022)
- [Unsupervised learning for cross-domain medical image synthesis using deformation invariant cycle consistency networks](https://mlsanity.com/paper/1808.03944/) (2018)
- [Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study](https://mlsanity.com/paper/2103.16262/) (2021)
- [Visual enhancement of Cone-beam CT by use of CycleGAN](https://mlsanity.com/paper/1901.05773/) (2019)

## Built on by (most cited first, 5 of 5)

- [Synthesis-based Imaging-Differentiation Representation Learning for Multi-Sequence 3D/4D MRI](https://mlsanity.com/paper/2302.00517/) (2023)
- [StainDiffuser: MultiTask Dual Diffusion Model for Virtual Staining](https://mlsanity.com/paper/2403.11340/) (2024)
- [Boosting Medical Image Synthesis via Registration-guided Consistency and Disentanglement Learning](https://mlsanity.com/paper/2407.07660/) (2024)
- [New multimodal similarity measure for image registration via modeling local functional dependence with linear combination of learned basis functions](https://mlsanity.com/paper/2503.05335/) (2025)
- [PRINTER:Deformation-Aware Adversarial Learning for Virtual IHC Staining with In Situ Fidelity](https://mlsanity.com/paper/2509.01214/) (2025)

## Abstract (arXiv)

Cross-modality image synthesis is an active research topic with multiple medical clinically relevant applications. Recently, methods allowing training with paired but misaligned data have started to emerge. However, no robust and well-performing methods applicable to a wide range of real world data sets exist. In this work, we propose a generic solution to the problem of cross-modality image synthesis with paired but non-aligned data by introducing new deformation equivariance encouraging loss functions. The method consists of joint training of an image synthesis network together with separate registration networks and allows adversarial training conditioned on the input even with misaligned data. The work lowers the bar for new clinical applications by allowing effortless training of cross-modality image synthesis networks for more difficult data sets.