mlsanityData to 7 Oct 2026

EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery

arXiv 2501.11347 ↗First posted 20 Jan 2025cs.CVcited by 37

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recently, Multimodal Large Language Models (MLLMs) have demonstrated their immense potential in computer-aided diagnosis and decision-making. In the context of robotic-assisted surgery, MLLMs can serve as effective tools for surgical training and guidance. However, there is still a lack of MLLMs specialized for surgical scene understanding in clinical applications. In this work, we introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding that surgeons encounter. To train our EndoChat, we construct the Surg-396K dataset through a novel pipeline that systematically extracts surgical information and generates structured annotations based on collected large-scale endoscopic surgery datasets. Furthermore, we introduce a multi-scale visual token interaction mechanism and a visual contrast-based reasoning mechanism to enhance the model's representation learning and reasoning capabilities. Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks. Additionally, we conduct evaluations with professional surgeons, most of whom provide positive feedback on collaborating with EndoChat. Overall, these results demonstrate that our EndoChat has great potential to significantly advance training and automation in robotic-assisted surgery.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 56 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

37 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug
  2. 2024In-context learning enables multimodal large language models to classify cancer pathology images12 Mar
  3. 2024LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery26 Feb
  4. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement21 Jan
  5. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  7. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks26 May

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  11. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  12. 2023Towards Generalist Biomedical AIcited by 286
18 more
  1. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  2. 2021Medical Visual Question Answering: A Surveycited by 93
  3. 2020Fast WordPiece Tokenizationcited by 85
  4. 2017Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answeringcited by 79
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  6. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  7. 2019BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detectioncited by 62
  8. 2021Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmarkcited by 58
  9. 2022Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformercited by 58
  10. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  11. 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
  12. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  13. 2023Advancing Surgical VQA with Scene Graph Knowledgecited by 41
  14. 2023CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 17
  15. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  16. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  17. 2024Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classificationcited by 8
  18. 2024Multitask Learning in Minimally Invasive Surgical Vision: A Reviewcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 20.

37 papers.

Sort:

By area:

  1. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  2. 2025SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysiscited by 15
  3. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  4. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  5. 2025How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessmentcited by 11
  6. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 8
  7. 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understandingcited by 8
  8. 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoningcited by 6
  9. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  10. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learningcited by 5
  11. 2025EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopycited by 3
  12. 2025Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgerycited by 3
25 more
  1. 2026Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planningcited by 3
  2. 2025Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AIcited by 2
  3. 2025BleedOrigin: Dynamic Bleeding Source Localization in Endoscopic Submucosal Dissection via Dual-Stage Detection and Trackingcited by 2
  4. 2025SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understandingcited by 2
  5. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validationcited by 2
  6. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?cited by 2
  7. 2025Zero-Shot Multi-modal Large Language Model v.s. Supervised Deep Learning: A Comparative Study on CT-Based Intracranial Hemorrhage Subtypingcited by 1
  8. 2025Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Modelscited by 1
  9. 2026SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Videocited by 1
  10. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  11. 2026SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomycited by 1
  12. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Modelscited by 1
  13. 2026Enhancing Medical Visual Grounding via Knowledge-guided Spatial Promptscited by 1
  14. 2026SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgerycited by 1
  15. 2025Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgerycited by 0
  16. 2025NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identificationcited by 0
  17. 2026Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learningcited by 0
  18. 2026SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmarkcited by 0
  19. 2026Sum-of-Checks: Structured Reasoning for Surgical Safety with Large Vision-Language Modelscited by 0
  20. 2026Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generationcited by 0
  21. 2026RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answeringcited by 0
  22. 2026Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgerycited by 0
  23. 2026Towards Grounded GI Endoscopy VQA via Multi-Task Learning on Small VLMscited by 0
  24. 2026Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMscited by 0
  25. 2026SurgNarrator: A Generative Retrieval Framework for Surgical Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
  2. 2026GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards
  3. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  4. 2024Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models
  5. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
  6. 2025Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgery
  7. 2025SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planning
  8. 2024GP-VLS: A general-purpose vision language model for surgery