mlsanityData to 7 Oct 2026

Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer

arXiv 2206.11053 ↗First posted 22 Jun 2022cs.CV cs.AI cs.LGcited by 58

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual question answering (VQA) in surgery is largely unexplored. Expert surgeons are scarce and are often overloaded with clinical and academic workloads. This overload often limits their time answering questionnaires from patients, medical students or junior residents related to surgical procedures. At times, students and junior residents also refrain from asking too many questions during classes to reduce disruption. While computer-aided simulators and recording of past surgical procedures have been made available for them to observe and improve their skills, they still hugely rely on medical experts to answer their questions. Having a Surgical-VQA system as a reliable 'second opinion' could act as a backup and ease the load on the medical experts in answering these questions. The lack of annotated medical data and the presence of domain-specific terms has limited the exploration of VQA for surgical procedures. In this work, we design a Surgical-VQA task that answers questionnaires on surgical procedures based on the surgical scene. Extending the MICCAI endoscopic vision challenge 2018 dataset and workflow recognition dataset further, we introduce two Surgical-VQA datasets with classification and sentence-based answers. To perform Surgical-VQA, we employ vision-text transformers models. We further introduce a residual MLP-based VisualBert encoder model that enforces interaction between visual and text tokens, improving performance in classification-based answering. Furthermore, we study the influence of the number of input image patches and temporal visual features on the model performance in both classification and sentence-based answering.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

16 of 29 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  7. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  8. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  9. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  10. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  11. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  12. 20202018 Robotic Scene Segmentation Challengecited by 123
4 more
  1. 2021Human-Adversarial Visual Question Answeringcited by 57
  2. 2020Learning and Reasoning with the Graph Structure Representation in Robotic Surgerycited by 29
  3. 2022Global-Reasoned Multi-Task Learning Model for Surgical Scene Understandingcited by 29
  4. 2021Visual Question Answering: which investigated applications?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 11; 2025: 22; 2026: 15.

58 papers.

Sort:

By area:

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  2. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  3. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  4. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  5. 2023Advancing Surgical VQA with Scene Graph Knowledgecited by 41
  6. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgerycited by 39
  7. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  8. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learningcited by 34
  9. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  10. 2024Kvasir-VQA: A Text-Image Pair GI Tract Datasetcited by 23
  11. 2024PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgerycited by 22
  12. 2025LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settingscited by 19
38 more
  1. 2023CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 17
  2. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  3. 2025OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understandingcited by 17
  4. 2024GP-VLS: A general-purpose vision language model for surgerycited by 16
  5. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  6. 2023Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgerycited by 15
  7. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMscited by 14
  8. 2024Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Modelscited by 13
  9. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  10. 2024VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgerycited by 12
  11. 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Frameworkcited by 11
  12. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  13. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  14. 2025SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splattingcited by 7
  15. 2025SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understandingcited by 7
  16. 2024Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquirycited by 6
  17. 2024Multitask Learning in Minimally Invasive Surgical Vision: A Reviewcited by 5
  18. 2025Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignmentcited by 5
  19. 2023A Survey on Image-text Multimodal Modelscited by 4
  20. 2025Text-driven Adaptation of Foundation Models for Few-shot Surgical Workflow Analysiscited by 3
  21. 2025When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQAcited by 3
  22. 2026Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motioncited by 3
  23. 2023Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesiscited by 2
  24. 2025DiN: Diffusion Model for Robust Medical VQA with Semantic Noisy Labelscited by 2
  25. 2025Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Modelscited by 2
  26. 2025SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understandingcited by 2
  27. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validationcited by 2
  28. 2026SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?cited by 2
  29. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  30. 2026SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomycited by 1
  31. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Modelscited by 1
  32. 2025Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgerycited by 0
  33. 2025Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaborationcited by 0
  34. 2025CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answeringcited by 0
  35. 2025SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attentioncited by 0
  36. 2025Generating Natural-Language Surgical Feedback: From Structured Representation to Domain-Grounded Evaluationcited by 0
  37. 2025Distilling Expert Surgical Knowledge: How to train local surgical VLMs for anatomy explanation in Complete Mesocolic Excisioncited by 0
  38. 2026TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answeringcited by 0

50 most cited of 58 All 58 (JSON)