# SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis arXiv: https://arxiv.org/abs/2506.07603 ยท HTML: https://mlsanity.com/paper/2506.07603/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 45 references that are in the map. - First posted on arXiv 9 Jun 2025. ## Builds on (most cited first, 23 of 23) - [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022) - [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023) - [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025) - [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016) - [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024) - [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023) - [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021) - [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024) - [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022) - [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023) - [PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection](https://mlsanity.com/paper/2502.12119/) (2025) - [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021) - [Foundation Models for Video Understanding: A Survey](https://mlsanity.com/paper/2405.03770/) (2024) - [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025) - [LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning](https://mlsanity.com/paper/2408.07981/) (2024) - [General surgery vision transformer: A video pre-trained foundation model for general surgery](https://mlsanity.com/paper/2403.05949/) (2024) - [TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration](https://mlsanity.com/paper/2505.17098/) (2025) - [Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition](https://mlsanity.com/paper/1812.00033/) (2018) - [EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos](https://mlsanity.com/paper/2405.19644/) (2024) - [LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings](https://mlsanity.com/paper/2503.19740/) (2025) - [Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning](https://mlsanity.com/paper/2505.17097/) (2025) - [Each Fake News is Fake in its Own Way: An Attribution Multi-Granularity Benchmark for Multimodal Fake News Detection](https://mlsanity.com/paper/2412.14686/) (2024) - [Event Recognition in Laparoscopic Gynecology Videos with Hybrid Transformers](https://mlsanity.com/paper/2312.00593/) (2023) ## Built on by (most cited first, 9 of 9) - [Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning](https://mlsanity.com/paper/2603.12787/) (2026) - [SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence](https://mlsanity.com/paper/2603.16822/) (2026) - [SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery](https://mlsanity.com/paper/2606.25905/) (2026) - [MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning](https://mlsanity.com/paper/2608.14015/) (2026) - [CoRe-DA: Contrastive Regression for Unsupervised Domain Adaptation in Surgical Skill Assessment](https://mlsanity.com/paper/2603.29666/) (2026) - [Scaling Video Pretraining for Surgical Foundation Models](https://mlsanity.com/paper/2603.29966/) (2026) - [A vision-language model and platform for temporally mapping surgery from video](https://mlsanity.com/paper/2603.22583/) (2026) - [SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark](https://mlsanity.com/paper/2604.20319/) (2026) - [Spatiotemporal Recognition of Abdominal Wall Layers During Optical Trocar Entry](https://mlsanity.com/paper/2610.14512/) (2026) ## Abstract (arXiv) Surgical video understanding is pivotal for enabling automated intraoperative decision-making, skill assessment, and postoperative quality improvement. However, progress in developing surgical video foundation models (FMs) remains hindered by the scarcity of large-scale, diverse datasets for pretraining and systematic evaluation. In this paper, we introduce SurgBench, a unified surgical video benchmarking framework comprising a pretraining dataset, SurgBench-P, and an evaluation benchmark, SurgBench-E. SurgBench offers extensive coverage of diverse surgical scenarios, with SurgBench-P encompassing 53 million frames across 22 surgical procedures and 11 specialties, and SurgBench-E providing robust evaluation across six categories (phase classification, camera motion, tool recognition, disease diagnosis, action classification, and organ detection) spanning 72 fine-grained tasks. Extensive experiments reveal that existing video FMs struggle to generalize across varied surgical video analysis tasks, whereas pretraining on SurgBench-P yields substantial performance improvements and superior cross-domain generalization to unseen procedures and modalities. Our dataset and code are available upon request.