# CS2: A Controllable and Simultaneous Synthesizer of Images and Annotations with Minimal Human Intervention

arXiv: https://arxiv.org/abs/2206.13394 · HTML: https://mlsanity.com/paper/2206.13394/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks](https://mlsanity.com/paper/1807.10225/) (2018)
- [Patch-Based Image Inpainting with Generative Adversarial Networks](https://mlsanity.com/paper/1803.07422/) (2018)
- [A New 2.5D Representation for Lymph Node Detection using Random Sets of Deep Convolutional Neural Network Observations](https://mlsanity.com/paper/1406.2639/) (2014)
- [Adversarial Synthesis Learning Enables Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1712.07695/) (2017)
- [Red blood cell image generation for data augmentation using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1901.06219/) (2019)

## Built on by (most cited first, 5 of 5)

- [Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imaging](https://mlsanity.com/paper/2308.06534/) (2023)
- [Less is More: Unsupervised Mask-guided Annotated CT Image Synthesis with Minimum Manual Segmentations](https://mlsanity.com/paper/2303.12747/) (2023)
- [Large-Kernel Attention for 3D Medical Image Segmentation](https://mlsanity.com/paper/2207.11225/) (2022)
- [3D Brain and Heart Volume Generative Models: A Survey](https://mlsanity.com/paper/2210.05952/) (2022)
- [Post-COVID Highlights: Challenges and Solutions of AI Techniques for Swift Identification of COVID-19](https://mlsanity.com/paper/2311.06258/) (2023)

## Abstract (arXiv)

The destitution of image data and corresponding expert annotations limit the training capacities of AI diagnostic models and potentially inhibit their performance. To address such a problem of data and label scarcity, generative models have been developed to augment the training datasets. Previously proposed generative models usually require manually adjusted annotations (e.g., segmentation masks) or need pre-labeling. However, studies have found that these pre-labeling based methods can induce hallucinating artifacts, which might mislead the downstream clinical tasks, while manual adjustment could be onerous and subjective. To avoid manual adjustment and pre-labeling, we propose a novel controllable and simultaneous synthesizer (dubbed CS2) in this study to generate both realistic images and corresponding annotations at the same time. Our CS2 model is trained and validated using high resolution CT (HRCT) data collected from COVID-19 patients to realize an efficient infections segmentation with minimal human intervention. Our contributions include 1) a conditional image synthesis network that receives both style information from reference CT images and structural information from unsupervised segmentation masks, and 2) a corresponding segmentation mask synthesis network to automatically segment these synthesized images simultaneously. Our experimental studies on HRCT scans collected from COVID-19 patients demonstrate that our CS2 model can lead to realistic synthesized datasets and promising segmentation results of COVID infections compared to the state-of-the-art nnUNet trained and fine-tuned in a fully supervised manner.