mlsanityData to 7 Oct 2026

GP-VLS: A general-purpose vision language model for surgery

arXiv 2407.19305 ↗First posted 27 Jul 2024cs.CV cs.LG q-bio.TOcited by 16

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Surgery requires comprehensive medical knowledge, visual assessment skills, and procedural expertise. While recent surgical AI models have focused on solving task-specific problems, there is a need for general-purpose systems that can understand surgical scenes and interact through natural language. This paper introduces GP-VLS, a general-purpose vision language model for surgery that integrates medical and surgical knowledge with visual scene understanding. For comprehensively evaluating general-purpose surgical models, we propose SurgiQual, which evaluates across medical and surgical knowledge benchmarks as well as surgical vision-language questions. To train GP-VLS, we develop six new datasets spanning medical knowledge, surgical textbooks, and vision-language pairs for tasks like phase recognition and tool identification. We show that GP-VLS significantly outperforms existing open- and closed-source models on surgical vision-language tasks, with 8-21% improvements in accuracy across SurgiQual benchmarks. GP-VLS also demonstrates strong performance on medical and surgical knowledge tests compared to open-source alternatives. Overall, GP-VLS provides an open-source foundation for developing AI assistants to support surgeons across a wide range of tasks and scenarios. The code and data for this work is publicly available at gpvls-surgery-vlm.github.io.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 27 Jul 2024.

Builds on

51 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery22 Mar
  2. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generation23 Nov

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  11. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  12. 2016YOLO9000: Better, Faster, Strongercited by 1,429
37 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  5. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  6. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  8. 2023Segment Anything in High Qualitycited by 373
  9. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  10. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  11. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  13. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  14. 2024Capabilities of Gemini Models in Medicinecited by 231
  15. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  16. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  17. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  18. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  19. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  20. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  21. 2022Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformercited by 58
  22. 2023SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challengecited by 54
  23. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  24. 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
  25. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  26. 2023Advancing Surgical VQA with Scene Graph Knowledgecited by 41
  27. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  28. 2024Addressing cognitive bias in medical language modelscited by 34
  29. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  30. 2024General surgery vision transformer: A video pre-trained foundation model for general surgerycited by 25
  31. 2022Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasetscited by 23
  32. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 19
  33. 2024ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexteritycited by 15
  34. 2023Intuitive Surgical SurgToolLoc and SurgVU Challenges Results: 2022-2025cited by 11
  35. 2023Language models are susceptible to incorrect patient self-diagnosis in medical applicationscited by 8
  36. 2023Surgical Gym: A high-performance GPU-based platform for reinforcement learning with surgical robotscited by 5
  37. 2024SuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistantscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  2. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  3. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  4. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  5. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  6. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  7. 2025EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgerycited by 3
  8. 2025SurgiSR4K: A High-Resolution Endoscopic Video Dataset for Robotic-Assisted Minimally Invasive Procedurescited by 2
  9. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  10. 2026Evaluating Large Vision-language Models for Surgical Tool Detectioncited by 1
  11. 2025Leveraging Generic Foundation Models for Multimodal Surgical Data Analysiscited by 0
  12. 2025Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaborationcited by 0
4 more
  1. 2025Generating Natural-Language Surgical Feedback: From Structured Representation to Domain-Grounded Evaluationcited by 0
  2. 2026Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learningcited by 0
  3. 2026RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answeringcited by 0
  4. 2026SurgNarrator: A Generative Retrieval Framework for Surgical Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  2. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
  3. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
  4. 2025Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
  5. 2025Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery
  6. 2025SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning
  7. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?
  8. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation