# Less is More: Unsupervised Mask-guided Annotated CT Image Synthesis with Minimum Manual Segmentations

arXiv: https://arxiv.org/abs/2303.12747 · HTML: https://mlsanity.com/paper/2303.12747/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Adversarial Learning for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/1802.07934/) (2018)
- [Synthetic Data Augmentation using GAN for Improved Liver Lesion Classification](https://mlsanity.com/paper/1801.02385/) (2018)
- [Superpixel Segmentation with Fully Convolutional Networks](https://mlsanity.com/paper/2003.12929/) (2020)
- [Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis](https://mlsanity.com/paper/2204.06929/) (2022)
- [Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT Image Augmentation for Object Detection](https://mlsanity.com/paper/1906.04962/) (2019)
- [Learning Data Augmentation for Brain Tumor Segmentation with Coarse-to-Fine Generative Adversarial Networks](https://mlsanity.com/paper/1805.11291/) (2018)
- [CS2: A Controllable and Simultaneous Synthesizer of Images and Annotations with Minimal Human Intervention](https://mlsanity.com/paper/2206.13394/) (2022)

## Built on by (most cited first, 8 of 8)

- [From Pixel to Cancer: Cellular Automata in Computed Tomography](https://mlsanity.com/paper/2403.06459/) (2024)
- [Is attention all you need in medical image analysis? A review](https://mlsanity.com/paper/2307.12775/) (2023)
- [Synthetic Data as Validation](https://mlsanity.com/paper/2310.16052/) (2023)
- [Beyond the Hype: A dispassionate look at vision-language models in medical scenario](https://mlsanity.com/paper/2408.08704/) (2024)
- [Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Models](https://mlsanity.com/paper/2609.24879/) (2026)
- [Post-COVID Highlights: Challenges and Solutions of AI Techniques for Swift Identification of COVID-19](https://mlsanity.com/paper/2311.06258/) (2023)
- [Advancing oncology with federated learning: transcending boundaries in breast, lung, and prostate cancer. A systematic review](https://mlsanity.com/paper/2408.05249/) (2024)
- [Deep Learning Approaches for Medical Imaging Under Varying Degrees of Label Availability: A Comprehensive Survey](https://mlsanity.com/paper/2504.11588/) (2025)

## Abstract (arXiv)

As a pragmatic data augmentation tool, data synthesis has generally returned dividends in performance for deep learning based medical image analysis. However, generating corresponding segmentation masks for synthetic medical images is laborious and subjective. To obtain paired synthetic medical images and segmentations, conditional generative models that use segmentation masks as synthesis conditions were proposed. However, these segmentation mask-conditioned generative models still relied on large, varied, and labeled training datasets, and they could only provide limited constraints on human anatomical structures, leading to unrealistic image features. Moreover, the invariant pixel-level conditions could reduce the variety of synthetic lesions and thus reduce the efficacy of data augmentation. To address these issues, in this work, we propose a novel strategy for medical image synthesis, namely Unsupervised Mask (UM)-guided synthesis, to obtain both synthetic images and segmentations using limited manual segmentation labels. We first develop a superpixel based algorithm to generate unsupervised structural guidance and then design a conditional generative model to synthesize images and annotations simultaneously from those unsupervised masks in a semi-supervised multi-task setting. In addition, we devise a multi-scale multi-task Fr'echet Inception Distance (MM-FID) and multi-scale multi-task standard deviation (MM-STD) to harness both fidelity and variety evaluations of synthetic CT images. With multiple analyses on different scales, we could produce stable image quality measurements with high reproducibility. Compared with the segmentation mask guided synthesis, our UM-guided synthesis provided high-quality synthetic images with significantly higher fidelity, variety, and utility (p<0.05 by Wilcoxon Signed Ranked test).