# Interactive Generation of Laparoscopic Videos with Diffusion Models

arXiv: https://arxiv.org/abs/2406.06537 · HTML: https://mlsanity.com/paper/2406.06537/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [ControlVideo: Training-free Controllable Text-to-Video Generation](https://mlsanity.com/paper/2305.13077/) (2023)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020)
- [RoentGen: Vision-Language Foundation Model for Chest X-ray Generation](https://mlsanity.com/paper/2211.12737/) (2022)
- [Diffusion Deformable Model for 4D Temporal Medical Image Generation](https://mlsanity.com/paper/2206.13295/) (2022)
- [On Aliased Resizing and Surprising Subtleties in GAN Evaluation](https://mlsanity.com/paper/2104.11222/) (2021)
- [Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation](https://mlsanity.com/paper/1907.02882/) (2019)
- [Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis](https://mlsanity.com/paper/2303.12644/) (2023)
- [Synthesising Rare Cataract Surgery Samples with Guided Diffusion Models](https://mlsanity.com/paper/2308.02587/) (2023)
- [Navigating the Synthetic Realm: Harnessing Diffusion-based Models for Laparoscopic Text-to-Image Generation](https://mlsanity.com/paper/2312.03043/) (2023)

## Built on by (most cited first, 8 of 8)

- [SurGen: Text-Guided Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2408.14028/) (2024)
- [SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation](https://mlsanity.com/paper/2412.14018/) (2024)
- [Ophora: A Large-Scale Data-Driven Text-Guided Ophthalmic Surgical Video Generation Model](https://mlsanity.com/paper/2505.07449/) (2025)
- [SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis](https://mlsanity.com/paper/2506.03082/) (2025)
- [SurGrID: Controllable Surgical Simulation via Scene Graph to Image Diffusion](https://mlsanity.com/paper/2502.07945/) (2025)
- [Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models](https://mlsanity.com/paper/2505.09858/) (2025)
- [Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model](https://mlsanity.com/paper/2510.07345/) (2025)
- [Surgical Video Generation From Diffusion to World Models: A Survey](https://mlsanity.com/paper/2608.26214/) (2026)

## Abstract (arXiv)

Generative AI, in general, and synthetic visual data generation, in specific, hold much promise for benefiting surgical training by providing photorealism to simulation environments. Current training methods primarily rely on reading materials and observing live surgeries, which can be time-consuming and impractical. In this work, we take a significant step towards improving the training process. Specifically, we use diffusion models in combination with a zero-shot video diffusion method to interactively generate realistic laparoscopic images and videos by specifying a surgical action through text and guiding the generation with tool positions through segmentation masks. We demonstrate the performance of our approach using the publicly available Cholec dataset family and evaluate the fidelity and factual correctness of our generated images using a surgical action recognition model as well as the pixel-wise F1-score for the spatial control of tool generation. We achieve an FID of 38.097 and an F1-score of 0.71.