# Endora: Video Generation Models as Endoscopy Simulators

arXiv: https://arxiv.org/abs/2403.11050 · HTML: https://mlsanity.com/paper/2403.11050/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 77 references that are in the map.
- First posted on arXiv 17 Mar 2024.

## Builds on (most cited first, 30 of 52)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [Generative Adversarial Network in Medical Imaging: A Review](https://mlsanity.com/paper/1809.07294/) (2018)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Temporal Generative Adversarial Nets with Singular Value Clipping](https://mlsanity.com/paper/1611.06624/) (2016)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Multi-scale self-guided attention for medical image segmentation](https://mlsanity.com/paper/1906.02849/) (2019)
- [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023)
- [On Aliased Resizing and Surprising Subtleties in GAN Evaluation](https://mlsanity.com/paper/2104.11222/) (2021)
- [Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding](https://mlsanity.com/paper/2012.05440/) (2020)

## Built on by (most cited first, 30 of 55)

- [U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation](https://mlsanity.com/paper/2406.02918/) (2024)
- [Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning](https://mlsanity.com/paper/2506.02327/) (2025)
- [LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction](https://mlsanity.com/paper/2406.16073/) (2024)
- [VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation](https://mlsanity.com/paper/2503.01739/) (2025)
- [SurGen: Text-Guided Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2408.14028/) (2024)
- [EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splatting](https://mlsanity.com/paper/2407.01029/) (2024)
- [Bora: Biomedical Generalist Video Generation Model](https://mlsanity.com/paper/2407.08944/) (2024)
- [DiffRect: Latent Diffusion Label Rectification for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2407.09918/) (2024)
- [SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation](https://mlsanity.com/paper/2412.14018/) (2024)
- [Event-guided Low-light Video Semantic Segmentation](https://mlsanity.com/paper/2411.00639/) (2024)
- [GaussianStego: A Generalizable Stenography Pipeline for Generative 3D Gaussians Splatting](https://mlsanity.com/paper/2407.01301/) (2024)
- [Analyzing Tumors by Synthesis](https://mlsanity.com/paper/2409.06035/) (2024)
- [How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment](https://mlsanity.com/paper/2511.01775/) (2025)
- [Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling](https://mlsanity.com/paper/2512.23162/) (2025)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [Surgical Vision World Model](https://mlsanity.com/paper/2503.02904/) (2025)
- [Ophora: A Large-Scale Data-Driven Text-Guided Ophthalmic Surgical Video Generation Model](https://mlsanity.com/paper/2505.07449/) (2025)
- [Polyp-Gen: Realistic and Diverse Polyp Image Generation for Endoscopic Dataset Expansion](https://mlsanity.com/paper/2501.16679/) (2025)
- [MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videos](https://mlsanity.com/paper/2507.05675/) (2025)
- [Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation](https://mlsanity.com/paper/2411.01647/) (2024)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2506.21287/) (2025)
- [Soft Masked Mamba Diffusion Model for CT to MRI Conversion](https://mlsanity.com/paper/2406.15910/) (2024)
- [GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation](https://mlsanity.com/paper/2407.05540/) (2024)
- [Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation](https://mlsanity.com/paper/2405.06948/) (2024)
- [When 3D Partial Points Meets SAM: Tooth Point Cloud Segmentation with Sparse Labels](https://mlsanity.com/paper/2409.01691/) (2024)
- [Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks](https://mlsanity.com/paper/2503.12531/) (2025)
- [Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification](https://mlsanity.com/paper/2409.17091/) (2024)
- [SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis](https://mlsanity.com/paper/2506.03082/) (2025)
- [FEAT: Full-Dimensional Efficient Attention Transformer for Medical Video Generation](https://mlsanity.com/paper/2506.04956/) (2025)

## Abstract (arXiv)

Generative models hold promise for revolutionizing medical education, robot-assisted surgery, and data augmentation for machine learning. Despite progress in generating 2D medical images, the complex domain of clinical video generation has largely remained untapped.This paper introduces model, an innovative approach to generate medical videos that simulate clinical endoscopy scenes. We present a novel generative model design that integrates a meticulously crafted spatial-temporal video transformer with advanced 2D vision foundation model priors, explicitly modeling spatial-temporal dynamics during video generation. We also pioneer the first public benchmark for endoscopy simulation with video generation models, adapting existing state-of-the-art methods for this endeavor.Endora demonstrates exceptional visual quality in generating endoscopy videos, surpassing state-of-the-art methods in extensive testing. Moreover, we explore how this endoscopy simulator can empower downstream video analysis tasks and even generate 3D medical scenes with multi-view consistency. In a nutshell, Endora marks a notable breakthrough in the deployment of generative AI for clinical endoscopy research, setting a substantial stage for further advances in medical content generation. For more details, please visit our project page: https://endora-medvidgen.github.io/.