mlsanityData to 14 Oct 2026

Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models

arXiv 2410.09750 ↗First posted 13 Oct 2024cs.CV cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Conversation agents powered by large language models are revolutionizing the way we interact with visual data. Recently, large vision-language models (LVLMs) have been extensively studied for both images and videos. However, these studies typically focus on common scenarios. In this work, we introduce an LVLM specifically designed for surgical scenarios. We integrate visual representations of surgical images and videos into the language feature space. Consequently, we establish a LVLM model, Surgical-LLaVA, fine-tuned on instruction following data of surgical scenarios. Our experiments demonstrate that Surgical-LLaVA exhibits impressive multi-modal chat abilities in surgical contexts, occasionally displaying multi-modal behaviors on unseen instructions. We conduct a quantitative evaluation of visual question-answering datasets for surgical scenarios. The results show superior performance compared to previous works, indicating the potential of our model to tackle more complex surgery scenarios.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

31 of 37 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
19 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2023Instruction Tuning with GPT-4cited by 686
  8. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 367
  9. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2024Capabilities of Gemini Models in Medicinecited by 237
  12. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 179
  13. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  14. 2020CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80cited by 106
  15. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  16. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  17. 2022Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformercited by 59
  18. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  19. 2022Towards Holistic Surgical Scene Understandingcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 7.

13 papers.

Sort:

By area:

  1. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  2. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  3. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  4. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  5. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgerycited by 6
  6. 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 4
  7. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  8. 2025SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understandingcited by 3
  9. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validationcited by 2
  10. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 2
  11. 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Videocited by 1
  12. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Modelscited by 1
1 more
  1. 2026Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videoscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning
  2. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
  3. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
  4. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  5. 2024GP-VLS: A general-purpose vision language model for surgery
  6. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
  7. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?
  8. 2024Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry