# SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation

arXiv: https://arxiv.org/abs/2412.14018 · HTML: https://mlsanity.com/paper/2412.14018/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [A Note on the Inception Score](https://mlsanity.com/paper/1801.01973/) (2018)
- [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://mlsanity.com/paper/2202.10571/) (2022)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model](https://mlsanity.com/paper/2405.20222/) (2024)
- [Endora: Video Generation Models as Endoscopy Simulators](https://mlsanity.com/paper/2403.11050/) (2024)
- [On the Content Bias in Fr'echet Video Distance](https://mlsanity.com/paper/2404.12391/) (2024)
- [A Recipe for Scaling up Text-to-Video Generation with Text-free Videos](https://mlsanity.com/paper/2312.15770/) (2023)
- [Self-Supervised Learning via Conditional Motion Propagation](https://mlsanity.com/paper/1903.11412/) (2019)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [SurGen: Text-Guided Diffusion Model for Surgical Video Generation](https://mlsanity.com/paper/2408.14028/) (2024)
- [Bora: Biomedical Generalist Video Generation Model](https://mlsanity.com/paper/2407.08944/) (2024)
- [CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection](https://mlsanity.com/paper/2410.07540/) (2024)
- [VISAGE: Video Synthesis using Action Graphs for Surgery](https://mlsanity.com/paper/2410.17751/) (2024)
- [Interactive Generation of Laparoscopic Videos with Diffusion Models](https://mlsanity.com/paper/2406.06537/) (2024)
- [Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation](https://mlsanity.com/paper/2411.01647/) (2024)
- [Surgical Text-to-Image Generation](https://mlsanity.com/paper/2407.09230/) (2024)

## Built on by (most cited first, 15 of 15)

- [How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment](https://mlsanity.com/paper/2511.01775/) (2025)
- [SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation](https://mlsanity.com/paper/2603.13024/) (2026)
- [Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks](https://mlsanity.com/paper/2503.12531/) (2025)
- [SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis](https://mlsanity.com/paper/2506.03082/) (2025)
- [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026)
- [Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement](https://mlsanity.com/paper/2505.01766/) (2025)
- [Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis](https://mlsanity.com/paper/2510.16371/) (2025)
- [SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics](https://mlsanity.com/paper/2606.19889/) (2026)
- [Towards Controllable Video Synthesis of Routine and Rare OR Events](https://mlsanity.com/paper/2602.21365/) (2026)
- [Real-Time Visual Obstruction Detection in Surgical Augmented Reality](https://mlsanity.com/paper/2608.00232/) (2026)
- [Surg-UniWorld: A Unified Surgical World Model with Multimodal Control Experts](https://mlsanity.com/paper/2608.06770/) (2026)
- [Surgical Video Generation From Diffusion to World Models: A Survey](https://mlsanity.com/paper/2608.26214/) (2026)
- [Multi-modal Representation Fusion for Long-term Microsurgical Anastomosis Trajectory Anticipation](https://mlsanity.com/paper/2610.14492/) (2026)
- [SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models](https://mlsanity.com/paper/2610.15645/) (2026)
- [Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets](https://mlsanity.com/paper/2610.17684/) (2026)

## Abstract (arXiv)

Surgical video generation can enhance medical education and research, but existing methods lack fine-grained motion control and realism. We introduce SurgSora, a framework that generates high-fidelity, motion-controllable surgical videos from a single input frame and user-specified motion cues. Unlike prior approaches that treat objects indiscriminately or rely on ground-truth segmentation masks, SurgSora leverages self-predicted object features and depth information to refine RGB appearance and optical flow for precise video synthesis. It consists of three key modules: (1) the Dual Semantic Injector, which extracts object-specific RGB-D features and segmentation cues to enhance spatial representations; (2) the Decoupled Flow Mapper, which fuses multi-scale optical flow with semantic features for realistic motion dynamics; and (3) the Trajectory Controller, which estimates sparse optical flow and enables user-guided object movement. By conditioning these enriched features within the Stable Video Diffusion, SurgSora achieves state-of-the-art visual authenticity and controllability in advancing surgical video synthesis, as demonstrated by extensive quantitative and qualitative comparisons. Our human evaluation in collaboration with expert surgeons further demonstrates the high realism of SurgSora-generated videos, highlighting the potential of our method for surgical training and education. Our project is available at https://surgsora.github.io/surgsora.github.io.