# Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

arXiv: https://arxiv.org/abs/1802.09655 · HTML: https://mlsanity.com/paper/1802.09655/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2018.

## Built on by (most cited first, 30 of 97)

- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models](https://mlsanity.com/paper/2209.12104/) (2022)
- [Source-Free Domain Adaptive Fundus Image Segmentation with Denoised Pseudo-Labeling](https://mlsanity.com/paper/2109.09735/) (2021)
- [UniMiSS: Universal Medical Self-Supervised Learning via Breaking Dimensionality Barrier](https://mlsanity.com/paper/2112.09356/) (2021)
- [Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation](https://mlsanity.com/paper/2112.09645/) (2021)
- [Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels](https://mlsanity.com/paper/2209.13476/) (2022)
- [BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys](https://mlsanity.com/paper/2310.10765/) (2023)
- [Unsupervised Domain Adaptation with Variational Approximation for Cardiac Segmentation](https://mlsanity.com/paper/2106.08752/) (2021)
- [DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Images](https://mlsanity.com/paper/2205.13723/) (2022)
- [LE-UDA: Label-efficient unsupervised domain adaptation for medical image segmentation](https://mlsanity.com/paper/2212.02078/) (2022)
- [Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization](https://mlsanity.com/paper/2102.06315/) (2021)
- [Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models](https://mlsanity.com/paper/2103.00429/) (2021)
- [COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training](https://mlsanity.com/paper/2203.16557/) (2022)
- [Multi-Modality Cardiac Image Computing: A Survey](https://mlsanity.com/paper/2208.12881/) (2022)
- [Structure-Preserving Synthesis: MaskGAN for Unpaired MR-CT Translation](https://mlsanity.com/paper/2307.16143/) (2023)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [Domain Adaptation Meets Zero-Shot Learning: An Annotation-Efficient Approach to Multi-Modality Medical Image Segmentation](https://mlsanity.com/paper/2203.10332/) (2022)
- [Mirror U-Net: Marrying Multimodal Fission with Multi-task Learning for Semantic Segmentation in Medical Imaging](https://mlsanity.com/paper/2303.07126/) (2023)
- [FakeNews: GAN-based generation of realistic 3D volumetric data -- A systematic review and taxonomy](https://mlsanity.com/paper/2207.01390/) (2022)
- [TarGAN: Target-Aware Generative Adversarial Networks for Multi-modality Medical Image Translation](https://mlsanity.com/paper/2105.08993/) (2021)
- [Adapt Everywhere: Unsupervised Adaptation of Point-Clouds and Entropy Minimisation for Multi-modal Cardiac Image Segmentation](https://mlsanity.com/paper/2103.08219/) (2021)
- [Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency](https://mlsanity.com/paper/2206.10571/) (2022)
- [SDC-UDA: Volumetric Unsupervised Domain Adaptation Framework for Slice-Direction Continuous Cross-Modality Medical Image Segmentation](https://mlsanity.com/paper/2305.11012/) (2023)
- [Dynamic Enhancement Network for Partial Multi-modality Person Re-identification](https://mlsanity.com/paper/2305.15762/) (2023)
- [METGAN: Generative Tumour Inpainting and Modality Synthesis in Light Sheet Microscopy](https://mlsanity.com/paper/2104.10993/) (2021)
- [GAN-based disentanglement learning for chest X-ray rib suppression](https://mlsanity.com/paper/2110.09134/) (2021)
- [Anatomy-guided Multimodal Registration by Learning Segmentation without Ground Truth: Application to Intraprocedural CBCT/MR Liver Segmentation and Registration](https://mlsanity.com/paper/2104.07056/) (2021)
- [Synthetic Data in Healthcare](https://mlsanity.com/paper/2304.03243/) (2023)

## Abstract (arXiv)

Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.