# Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis

arXiv: https://arxiv.org/abs/2204.06929 · HTML: https://mlsanity.com/paper/2204.06929/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [Freehand Ultrasound Image Simulation with Spatially-Conditioned Generative Adversarial Networks](https://mlsanity.com/paper/1707.05392/) (2017)
- [SkrGAN: Sketching-rendering Unconditional Generative Adversarial Networks for Medical Image Synthesis](https://mlsanity.com/paper/1908.04346/) (2019)
- [Simulating Patho-realistic Ultrasound Images using Deep Generative Networks with Adversarial Learning](https://mlsanity.com/paper/1712.07881/) (2017)
- [Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments](https://mlsanity.com/paper/2005.13178/) (2020)
- [Virtual PET Images from CT Data Using Deep Convolutional Networks: Initial Results](https://mlsanity.com/paper/1707.09585/) (2017)
- [SUSAN: Segment Unannotated image Structure using Adversarial Network](https://mlsanity.com/paper/1812.00555/) (2018)
- [Synthesis and Edition of Ultrasound Images via Sketch Guided Progressive Growing GANs](https://mlsanity.com/paper/2004.00226/) (2020)

## Built on by (most cited first, 30 of 42)

- [Segment Anything Model for Medical Images?](https://mlsanity.com/paper/2304.14660/) (2023)
- [Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis](https://mlsanity.com/paper/2303.12644/) (2023)
- [Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges](https://mlsanity.com/paper/2407.00116/) (2024)
- [EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharing](https://mlsanity.com/paper/2406.00808/) (2024)
- [Synthetic optical coherence tomography angiographs for detailed retinal vessel segmentation without human annotations](https://mlsanity.com/paper/2306.10941/) (2023)
- [Derm-T2IM: Harnessing Synthetic Skin Lesion Data via Stable Diffusion Models for Enhanced Skin Disease Classification using ViT and CNN](https://mlsanity.com/paper/2401.05159/) (2024)
- [Fourier Test-time Adaptation with Multi-level Consistency for Robust Classification](https://mlsanity.com/paper/2306.02544/) (2023)
- [Online Reflective Learning for Robust Medical Image Segmentation](https://mlsanity.com/paper/2207.00476/) (2022)
- [Less is More: Unsupervised Mask-guided Annotated CT Image Synthesis with Minimum Manual Segmentations](https://mlsanity.com/paper/2303.12747/) (2023)
- [Echocardiography video synthesis from end diastolic semantic map via diffusion model](https://mlsanity.com/paper/2310.07131/) (2023)
- [Weakly-supervised High-fidelity Ultrasound Video Synthesis with Feature Decoupling](https://mlsanity.com/paper/2207.00474/) (2022)
- [Training-Free Condition Video Diffusion Models for single frame Spatial-Semantic Echocardiogram Synthesis](https://mlsanity.com/paper/2408.03035/) (2024)
- [FetalFlex: Anatomy-Guided Diffusion Model for Flexible Control on Fetal Ultrasound Image Synthesis](https://mlsanity.com/paper/2503.14906/) (2025)
- [EchoFlow: A Foundation Model for Cardiac Ultrasound Image and Video Generation](https://mlsanity.com/paper/2503.22357/) (2025)
- [Fine-grained Correlation Loss for Regression](https://mlsanity.com/paper/2207.00347/) (2022)
- [DermSynth3D: Synthesis of in-the-wild Annotated Dermatology Images](https://mlsanity.com/paper/2305.12621/) (2023)
- [Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification](https://mlsanity.com/paper/2409.17091/) (2024)
- [Lung-DDPM+: Efficient Thoracic CT Image Synthesis using Diffusion Probabilistic Model](https://mlsanity.com/paper/2508.09327/) (2025)
- [Agent with Tangent-based Formulation and Anatomical Perception for Standard Plane Localization in 3D Ultrasound](https://mlsanity.com/paper/2207.00475/) (2022)
- [Inflated 3D Convolution-Transformer for Weakly-supervised Carotid Stenosis Grading with Ultrasound Videos](https://mlsanity.com/paper/2306.02548/) (2023)
- [Flip Learning: Weakly Supervised Erase to Segment Nodules in Breast Ultrasound](https://mlsanity.com/paper/2503.20685/) (2025)
- [MoNetV2: Enhanced Motion Network for Freehand 3D Ultrasound Reconstruction](https://mlsanity.com/paper/2506.15835/) (2025)
- [Artificial Intelligence for Detecting Fetal Orofacial Clefts and Advancing Medical Education](https://mlsanity.com/paper/2603.06522/) (2026)
- [A Comparative Study of Gastric Histopathology Sub-size Image Classification: from Linear Regression to Visual Transformer](https://mlsanity.com/paper/2205.12843/) (2022)
- [High-Quality Medical Image Generation from Free-hand Sketch](https://mlsanity.com/paper/2402.00353/) (2024)
- [MEDDAP: Medical Dataset Enhancement via Diversified Augmentation Pipeline](https://mlsanity.com/paper/2403.16335/) (2024)
- [S-CycleGAN: Semantic Segmentation Enhanced CT-Ultrasound Image-to-Image Translation for Robotic Ultrasonography](https://mlsanity.com/paper/2406.01191/) (2024)
- [CSG: A Context-Semantic Guided Diffusion Approach in De Novo Musculoskeletal Ultrasound Image Generation](https://mlsanity.com/paper/2412.05833/) (2024)
- [Uncertainty-aware Diffusion and Reinforcement Learning for Joint Plane Localization and Anomaly Diagnosis in 3D Ultrasound](https://mlsanity.com/paper/2506.23538/) (2025)
- [OnUVS: Online Feature Decoupling Framework for High-Fidelity Ultrasound Video Synthesis](https://mlsanity.com/paper/2308.08269/) (2023)

## Abstract (arXiv)

Ultrasound (US) imaging is widely used for anatomical structure inspection in clinical diagnosis. The training of new sonographers and deep learning based algorithms for US image analysis usually requires a large amount of data. However, obtaining and labeling large-scale US imaging data are not easy tasks, especially for diseases with low incidence. Realistic US image synthesis can alleviate this problem to a great extent. In this paper, we propose a generative adversarial network (GAN) based image synthesis framework. Our main contributions include: 1) we present the first work that can synthesize realistic B-mode US images with high-resolution and customized texture editing features; 2) to enhance structural details of generated images, we propose to introduce auxiliary sketch guidance into a conditional GAN. We superpose the edge sketch onto the object mask and use the composite mask as the network input; 3) to generate high-resolution US images, we adopt a progressive training strategy to gradually generate high-resolution images from low-resolution images. In addition, a feature loss is proposed to minimize the difference of high-level features between the generated and real images, which further improves the quality of generated images; 4) the proposed US image synthesis method is quite universal and can also be generalized to the US images of other anatomical structures besides the three ones tested in our study (lung, hip joint, and ovary); 5) extensive experiments on three large US image datasets are conducted to validate our method. Ablation studies, customized texture editing, user studies, and segmentation tests demonstrate promising results of our method in synthesizing realistic US images.