mlsanityData to 14 Oct 2026

SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planning

arXiv 2506.07196 ↗First posted 8 Jun 2025cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Effective evaluation is critical for driving advancements in MLLM research. The surgical action planning (SAP) task, which aims to generate future action sequences from visual inputs, demands precise and sophisticated analytical capabilities. Unlike mathematical reasoning, surgical decision-making operates in life-critical domains and requires meticulous, verifiable processes to ensure reliability and patient safety. This task demands the ability to distinguish between atomic visual actions and coordinate complex, long-horizon procedures, capabilities that are inadequately evaluated by current benchmarks. To address this gap, we introduce SAP-Bench, a large-scale, high-quality dataset designed to enable multimodal large language models (MLLMs) to perform interpretable surgical action planning. Our SAP-Bench benchmark, derived from the cholecystectomy procedures context with the mean duration of 1137.5s, and introduces temporally-grounded surgical action annotations, comprising the 1,226 clinically validated action clips (mean duration: 68.7s) capturing five fundamental surgical actions across 74 procedures. The dataset provides 1,152 strategically sampled current frames, each paired with the corresponding next action as multimodal analysis anchors. We propose the MLLM-SAP framework that leverages MLLMs to generate next action recommendations from the current surgical scene and natural language instructions, enhanced with injected surgical domain knowledge. To assess our dataset's effectiveness and the broader capabilities of current models, we evaluate seven state-of-the-art MLLMs (e.g., OpenAI-o1, GPT-4o, QwenVL2.5-72B, Claude-3.5-Sonnet, GeminiPro2.5, Step-1o, and GLM-4v) and reveal critical gaps in next action prediction performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 29 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

24 of 29 references.

References · 24

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  8. 2022Visual Spatial Reasoningcited by 293
  9. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  10. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 141
  11. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  12. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
12 more
  1. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 110
  2. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  3. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  4. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  5. 2021Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmarkcited by 60
  6. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  7. 2024Pixel-Wise Recognition for Holistic Surgical Scene Understandingcited by 49
  8. 2023PALM: Predicting Actions through Language Modelscited by 29
  9. 2023Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023cited by 16
  10. 2025SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysiscited by 15
  11. 2024CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Modelscited by 11
  12. 2024Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgerycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
  2. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
  3. 2026SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics
  4. 2026Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning
  5. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model
  6. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence
  7. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
  8. 2026GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards