# SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding

arXiv: https://arxiv.org/abs/2511.03325 · HTML: https://mlsanity.com/paper/2511.03325/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 18 of 18)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer](https://mlsanity.com/paper/2206.11053/) (2022)
- [SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery](https://mlsanity.com/paper/2304.09974/) (2023)
- [Advancing Surgical VQA with Scene Graph Knowledge](https://mlsanity.com/paper/2312.10251/) (2023)
- [Kvasir-VQA: A Text-Image Pair GI Tract Dataset](https://mlsanity.com/paper/2409.01437/) (2024)
- [REAL-Colon: A dataset for developing real-world AI applications in colonoscopy](https://mlsanity.com/paper/2403.02163/) (2024)
- [PitVQA++: Vector Matrix-Low-Rank Adaptation for Open-Ended Visual Question Answering in Pituitary Surgery](https://mlsanity.com/paper/2502.14149/) (2025)

## Built on by (most cited first, 7 of 7)

- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering](https://mlsanity.com/paper/2603.09696/) (2026)
- [Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting](https://mlsanity.com/paper/2604.21082/) (2026)
- [SurgLQA: Scalable Long-Horizon Surgical Video Question Answering](https://mlsanity.com/paper/2605.17915/) (2026)
- [Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA](https://mlsanity.com/paper/2606.17279/) (2026)
- [Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA](https://mlsanity.com/paper/2607.17834/) (2026)
- [STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs](https://mlsanity.com/paper/2609.08543/) (2026)

## Abstract (arXiv)

Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation. We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video--Text Encoder to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions, which a fine-tuned large language model (LLM) then decodes into coherent answers. To evaluate its performance, we curated REAL-Colon-VQA, a colonoscopic video dataset that includes motion-related questions and diagnostic attributes, as well as out-of-template questions with rephrased or semantically altered formulations to assess model robustness. Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models, particularly in keyword accuracy, improving over PitVQA by +11% on REAL-Colon-VQA and +9% on EndoVis18-VQA. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing. SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally-aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts more effectively. Code and dataset available at https://github.com/madratak/SurgViVQA.