mlsanityData to 7 Oct 2026

General surgery vision transformer: A video pre-trained foundation model for general surgery

arXiv 2403.05949 ↗First posted 9 Mar 2024cs.CV cs.LG q-bio.TOcited by 25

Not on the map. Why?

Abstract (arXiv)

The absence of openly accessible data and specialized foundation models is a major barrier for computational research in surgery. Toward this, (i) we open-source the largest dataset of general surgery videos to-date, consisting of 680 hours of surgical videos, including data from robotic and laparoscopic techniques across 28 procedures; (ii) we propose a technique for video pre-training a general surgery vision transformer (GSViT) on surgical videos based on forward video prediction that can run in real-time for surgical applications, toward which we open-source the code and weights of GSViT; (iii) we also release code and weights for procedure-specific fine-tuned versions of GSViT across 10 procedures; (iv) we demonstrate the performance of GSViT on the Cholec80 phase annotation task, displaying improved performance over state-of-the-art single frame predictors.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

24 of 32 references.

References · 24

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2016Layer Normalizationcited by 5,579
  3. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  7. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  8. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  9. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  10. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  11. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  12. 2020TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networkscited by 139
12 more
  1. 20202018 Robotic Scene Segmentation Challengecited by 123
  2. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  3. 2020CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80cited by 104
  4. 2019Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysiscited by 75
  5. 2023LoViT: Long Video Transformer for Surgical Phase Recognitioncited by 56
  6. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  7. 2024Addressing cognitive bias in medical language modelscited by 34
  8. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  9. 2023ViT-AE++: Improving Vision Transformer Autoencoder for Self-supervised Medical Image Representationscited by 12
  10. 2023Intuitive Surgical SurgToolLoc and SurgVU Challenges Results: 2022-2025cited by 11
  11. 2023Whether and When does Endoscopy Domain Pretraining Make Sense?cited by 8
  12. 2023Language models are susceptible to incorrect patient self-diagnosis in medical applicationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 12.

25 papers.

Sort:

By area:

  1. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  2. 2025LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settingscited by 19
  3. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16
  4. 2025SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learningcited by 16
  5. 2025SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysiscited by 15
  6. 2024VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgerycited by 12
  7. 2025Large-scale Self-supervised Video Foundation Model for Intelligent Surgerycited by 10
  8. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 8
  9. 2025The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessmentcited by 6
  10. 2025SciVid: Cross-Domain Evaluation of Video Models in Scientific Applicationscited by 5
  11. 2026SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videoscited by 4
  12. 2026CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognitioncited by 3
13 more
  1. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  2. 2025COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasetscited by 1
  3. 2025Analysis of Transferability Estimation Metrics for Surgical Phase Recognitioncited by 1
  4. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  5. 2026Surgical Anatomy Recognition with Context Learning using Foundation Representationscited by 1
  6. 2026SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgerycited by 1
  7. 2026HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Spacecited by 1
  8. 2025Generating Natural-Language Surgical Feedback: From Structured Representation to Domain-Grounded Evaluationcited by 0
  9. 2026On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-trainingcited by 0
  10. 2026A vision-language model and platform for temporally mapping surgery from videocited by 0
  11. 2026SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmarkcited by 0
  12. 2026DART: Depth-as-Target Pretraining for Surgical Vision Foundation Modelscited by 0
  13. 2026A Surgical Foundation Model Reveals Task-Dependent Label Efficiencycited by 0