# Generative deep learning for foundational video translation in ultrasound

arXiv: https://arxiv.org/abs/2511.03255 · HTML: https://mlsanity.com/paper/2511.03255/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [How good is my GAN?](https://mlsanity.com/paper/1807.09499/) (2018)
- [Deep Learning-based Image and Video Inpainting: A Survey](https://mlsanity.com/paper/2401.03395/) (2024)
- [Echo from noise: synthetic ultrasound image generation using diffusion models for real image segmentation](https://mlsanity.com/paper/2305.05424/) (2023)
- [OCR-VQGAN: Taming Text-within-Image Generation](https://mlsanity.com/paper/2210.11248/) (2022)
- [FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysis](https://mlsanity.com/paper/2502.14807/) (2025)
- [Similarity and Quality Metrics for MR Image-To-Image Translation](https://mlsanity.com/paper/2405.08431/) (2024)
- [A study of why we need to reassess full reference image quality assessment with medical images](https://mlsanity.com/paper/2405.19097/) (2024)
- [Adversarial Inpainting of Medical Image Modalities](https://mlsanity.com/paper/1810.06621/) (2018)
- [Label-free segmentation from cardiac ultrasound using self-supervised learning](https://mlsanity.com/paper/2210.04979/) (2022)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [Domain-guided data augmentation for deep learning on medical imaging](https://mlsanity.com/paper/2210.04977/) (2022)
- [Beyond Size and Class Balance: Alpha as a New Dataset Quality Metric for Deep Learning](https://mlsanity.com/paper/2407.15724/) (2024)
- [Are foundation models efficient for medical image segmentation?](https://mlsanity.com/paper/2311.04847/) (2023)

## Abstract (arXiv)

Deep learning (DL) has the potential to revolutionize image acquisition and interpretation across medicine, however, attention to data imbalance and missingness is required. Ultrasound data presents a particular challenge because in addition to different views and structures, it includes several sub-modalities-such as greyscale and color flow doppler (CFD)-that are often imbalanced in clinical studies. Image translation can help balance datasets but is challenging for ultrasound sub-modalities to date. Here, we present a generative method for ultrasound CFD-greyscale video translation, trained on 54,975 videos and tested on 8,368. The method developed leveraged pixel-wise, adversarial, and perceptual loses and utilized two networks: one for reconstructing anatomic structures and one for denoising to achieve realistic ultrasound imaging. Average pairwise SSIM between synthetic videos and ground truth was 0.91+/-0.04. Synthetic videos performed indistinguishably from real ones in DL classification and segmentation tasks and when evaluated by blinded clinical experts: F1 score was 0.9 for real and 0.89 for synthetic videos; Dice score between real and synthetic segmentation was 0.97. Overall clinician accuracy in distinguishing real vs synthetic videos was 54+/-6% (42-61%), indicating realistic synthetic videos. Although trained only on heart videos, the model worked well on ultrasound spanning several clinical domains (average SSIM 0.91+/-0.05), demonstrating foundational abilities. Together, these data expand the utility of retrospectively collected imaging and augment the dataset design toolbox for medical imaging.