# SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

arXiv: https://arxiv.org/abs/2606.25905 · HTML: https://mlsanity.com/paper/2606.25905/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020)
- [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition](https://mlsanity.com/paper/2405.10075/) (2024)
- [Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery](https://mlsanity.com/paper/2312.11250/) (2023)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [The TUM LapChole dataset for the M2CAI 2016 workflow challenge](https://mlsanity.com/paper/1610.09278/) (2016)
- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence](https://mlsanity.com/paper/2506.02555/) (2025)
- [General surgery vision transformer: A video pre-trained foundation model for general surgery](https://mlsanity.com/paper/2403.05949/) (2024)
- [PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery](https://mlsanity.com/paper/2405.13949/) (2024)
- [The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark](https://mlsanity.com/paper/2312.12429/) (2023)
- [EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos](https://mlsanity.com/paper/2405.19644/) (2024)
- [LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings](https://mlsanity.com/paper/2503.19740/) (2025)
- [SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning](https://mlsanity.com/paper/2509.10555/) (2025)
- [Surgical Visual Understanding (SurgVU) Dataset](https://mlsanity.com/paper/2501.09209/) (2025)
- [Large-scale Self-supervised Video Foundation Model for Intelligent Surgery](https://mlsanity.com/paper/2506.02692/) (2025)
- [SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis](https://mlsanity.com/paper/2506.07603/) (2025)
- [Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)](https://mlsanity.com/paper/2511.06549/) (2025)
- [SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning](https://mlsanity.com/paper/2603.06570/) (2026)
- [SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding](https://mlsanity.com/paper/2509.00357/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation](https://mlsanity.com/paper/2610.09800/) (2026)
- [Spatiotemporal Recognition of Abdominal Wall Layers During Optical Trocar Entry](https://mlsanity.com/paper/2610.14512/) (2026)

## Abstract (arXiv)

We introduce SurgAtlas, the largest surgical video-language dataset to date, comprising 15,291 videos (2,391 hours) spanning 18 surgical specialties and over 5,000 procedure types, sourced entirely from publicly available YouTube content. SurgAtlas is also the first surgical video-language dataset to include open surgery at scale, with 6,182 open procedure videos alongside over 9,000 minimally invasive recordings, and the first to establish standardized benchmarks for open-surgery video understanding. We additionally provide an expert-validated subset with verified visual question-answer pairs across diverse open and minimally invasive procedures, serving as a clinically grounded benchmark for surgical reasoning. Compared with existing surgical video-language datasets, SurgAtlas provides one of the most diverse annotation schemas, combining segment-level captions, step- and phase-level descriptions, video-level surgical descriptions, and reasoning-oriented question-answer pairs organized within a hierarchical taxonomy. These annotations are constructed through an automated multi-tier pipeline with LLM-based enrichment and a staged VQA generation framework with explicit groundedness verification. The scale and diversity of SurgAtlas enable training surgical foundation models with broad procedural coverage: we finetune Qwen3-VL-8B through a two-stage captioning-then-instruction pipeline and achieve competitive or state-of-the-art results on multiple established surgical benchmarks, including phase recognition, triplet detection, and reasoning question answering. More broadly, SurgAtlas provides a large native public video corpus that can support future large-scale pretraining of multimodal surgical AI systems and contribute to the development of next-generation foundation models for surgery. Dataset publicly available at https://huggingface.co/datasets/filbel/SurgAtlas