mlsanityData to 7 Oct 2026

Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery

arXiv 2405.10948 ↗First posted 22 Mar 2024cs.CV cs.AI cs.ROcited by 39

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent advancements in Surgical Visual Question Answering (Surgical-VQA) and related region grounding have shown great promise for robotic and medical applications, addressing the critical need for automated methods in personalized surgical mentorship. However, existing models primarily provide simple structured answers and struggle with complex scenarios due to their limited capability in recognizing long-range dependencies and aligning multimodal information. In this paper, we introduce Surgical-LVLM, a novel personalized large vision-language model tailored for complex surgical scenarios. Leveraging the pre-trained large vision-language model and specialized Visual Perception LoRA (VP-LoRA) blocks, our model excels in understanding complex visual-language tasks within surgical contexts. In addressing the visual grounding task, we propose the Token-Interaction (TIT) module, which strengthens the interaction between the grounding module and the language responses of the Large Visual Language Model (LVLM) after projecting them into the latent space. We demonstrate the effectiveness of Surgical-LVLM on several benchmarks, including EndoVis-17-VQLA, EndoVis-18-VQLA, and a newly introduced EndoVis Conversations dataset, which sets new performance standards. Our work contributes to advancing the field of automated surgical mentorship by providing a context-aware solution.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 41 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

37 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
24 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2024VMamba: Visual State Space Modelcited by 955
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  7. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  9. 2022Focal Modulation Networkscited by 186
  10. 20192017 Robotic Instrument Segmentation Challengecited by 166
  11. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  12. 20202018 Robotic Scene Segmentation Challengecited by 123
  13. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  14. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  15. 2019BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detectioncited by 62
  16. 2022Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformercited by 58
  17. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  18. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  19. 2023Advancing Surgical VQA with Scene Graph Knowledgecited by 41
  20. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31
  21. 2020Learning and Reasoning with the Graph Structure Representation in Robotic Surgerycited by 29
  22. 2022Global-Reasoned Multi-Task Learning Model for Surgical Scene Understandingcited by 29
  23. 2023CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 17
  24. 2023Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgerycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 18.

39 papers.

Sort:

By area:

  1. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  2. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  3. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  4. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  5. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  6. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16
  7. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  8. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
  9. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  10. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  11. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  12. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
27 more
  1. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 6
  2. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Modelscited by 5
  3. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learningcited by 5
  4. 2025Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglementcited by 3
  5. 2026WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Modelscited by 3
  6. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validationcited by 2
  7. 2025A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Modelscited by 1
  8. 2025EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgerycited by 1
  9. 2026Benchmarking Direct Preference Optimization for Medical Large Vision-Language Modelscited by 1
  10. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  11. 2026Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysiscited by 1
  12. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generationcited by 1
  13. 2024V2-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopycited by 0
  14. 2025Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgerycited by 0
  15. 2025Generating Natural-Language Surgical Feedback: From Structured Representation to Domain-Grounded Evaluationcited by 0
  16. 2025NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identificationcited by 0
  17. 2026A 4D Representation for Training-Free Agentic Reasoning from Monocular Laparoscopic Videocited by 0
  18. 2026SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmarkcited by 0
  19. 2026LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Modelscited by 0
  20. 2026Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQAcited by 0
  21. 2026Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgerycited by 0
  22. 2026Towards Robustness against Typographic Attack with Training-free Concept Localizationcited by 0
  23. 2026DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splattingcited by 0
  24. 2026Real-Time Visual Obstruction Detection in Surgical Augmented Realitycited by 0
  25. 2026Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMscited by 0
  26. 2026SurgNarrator: A Generative Retrieval Framework for Surgical Video Understandingcited by 0
  27. 2026A Comprehensive Review of Generative Physical Artificial Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery
  2. 2024Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models
  3. 2024LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery
  4. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?
  5. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  6. 2025Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgery
  7. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning
  8. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering