# SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models arXiv: https://arxiv.org/abs/2412.02332 ยท HTML: https://mlsanity.com/paper/2412.02332/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. - Builds on 50 of its 68 references that are in the map. - First posted on arXiv 3 Dec 2024. ## Placed in - Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/) ## Builds on (most cited first, 30 of 50) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019) - [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023) - [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018) - [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018) - [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023) - [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015) - [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023) - [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023) - [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023) - [Diffusion Models for Medical Anomaly Detection](https://mlsanity.com/paper/2203.04306/) (2022) - [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023) - [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022) - [MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model](https://mlsanity.com/paper/2211.00611/) (2022) - [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022) - [InstanceDiffusion: Instance-level Control for Image Generation](https://mlsanity.com/paper/2402.03290/) (2024) - [ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback](https://mlsanity.com/paper/2404.07987/) (2024) - [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021) - [MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis](https://mlsanity.com/paper/2402.05408/) (2024) - [HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation](https://mlsanity.com/paper/2304.04269/) (2023) - [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2207.08518/) (2022) - [Ambiguous Medical Image Segmentation using Diffusion Models](https://mlsanity.com/paper/2304.04745/) (2023) - [DOLCE: A Model-Based Probabilistic Diffusion Framework for Limited-Angle CT Reconstruction](https://mlsanity.com/paper/2211.12340/) (2022) - [Temporal Memory Relation Network for Workflow Recognition from Surgical Video](https://mlsanity.com/paper/2103.16327/) (2021) - [Diffusion Deformable Model for 4D Temporal Medical Image Generation](https://mlsanity.com/paper/2206.13295/) (2022) - [CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2303.14081/) (2023) ## Built on by (most cited first, 8 of 8) - [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026) - [Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models](https://mlsanity.com/paper/2505.09858/) (2025) - [Endoscopic Depth Estimation Based on Deep Learning: A Survey](https://mlsanity.com/paper/2507.20881/) (2025) - [Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model](https://mlsanity.com/paper/2510.07345/) (2025) - [SCARED-C: Corrected Camera Poses for Endoscopic Depth Estimation](https://mlsanity.com/paper/2605.16628/) (2026) - [Clinically-guided Data Synthesis for Laryngeal Lesion Detection](https://mlsanity.com/paper/2508.06182/) (2025) - [Towards Application Aligned Synthetic Surgical Image Synthesis](https://mlsanity.com/paper/2509.18796/) (2025) - [ASTRA-Net: Anatomy-Specific Transfer and Representation Alignment for Drug-Induced Sleep Endoscopy Segmentation](https://mlsanity.com/paper/2607.21370/) (2026) ## Abstract (arXiv) Computer-assisted surgical (CAS) systems enhance surgical execution and outcomes by providing advanced support to surgeons. These systems often rely on deep learning models trained on complex, challenging-to-annotate data. While synthetic data generation can address these challenges, enhancing the realism of such data is crucial. This work introduces a multi-stage pipeline for generating realistic synthetic data, featuring a fully-fledged surgical simulator that automatically produces all necessary annotations for modern CAS systems. This simulator generates a wide set of annotations that surpass those available in public synthetic datasets. Additionally, it offers a more complex and realistic simulation of surgical interactions, including the dynamics between surgical instruments and deformable anatomical environments, outperforming existing approaches. To further bridge the visual gap between synthetic and real data, we propose a lightweight and flexible image-to-image translation method based on Stable Diffusion (SD) and Low-Rank Adaptation (LoRA). This method leverages a limited amount of annotated data, enables efficient training, and maintains the integrity of annotations generated by our simulator. The proposed pipeline is experimentally validated and can translate synthetic images into images with real-world characteristics, which can generalize to real-world context, thereby improving both training and CAS guidance. The code and the dataset are available at https://github.com/SanoScience/SimuScope.