# TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering

arXiv: https://arxiv.org/abs/2603.09696 · HTML: https://mlsanity.com/paper/2603.09696/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer](https://mlsanity.com/paper/2206.11053/) (2022)
- [SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery](https://mlsanity.com/paper/2304.09974/) (2023)
- [Advancing Surgical VQA with Scene Graph Knowledge](https://mlsanity.com/paper/2312.10251/) (2023)
- [Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?](https://mlsanity.com/paper/2404.18624/) (2024)
- [PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery](https://mlsanity.com/paper/2405.13949/) (2024)
- [REAL-Colon: A dataset for developing real-world AI applications in colonoscopy](https://mlsanity.com/paper/2403.02163/) (2024)
- [SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding](https://mlsanity.com/paper/2511.03325/) (2025)
- [Questioning the Stability of Visual Question Answering](https://mlsanity.com/paper/2511.11206/) (2025)
- [When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQA](https://mlsanity.com/paper/2511.01458/) (2025)
- [Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning](https://mlsanity.com/paper/2508.18687/) (2025)

## Abstract (arXiv)

Surgical Video Question Answering (VideoQA) requires accurate temporal grounding while remaining robust to natural variation in how clinicians phrase questions, where linguistic bias can arise. Standard Parameter Efficient Fine Tuning (PEFT) methods adapt pretrained projections without explicitly modeling frame-to-frame interactions within the adaptation pathway, limiting their ability to exploit sparse temporal evidence. We introduce TemporalDoRA, a video-specific PEFT formulation that extends Weight-Decomposed Low-Rank Adaptation by (i) inserting lightweight temporal Multi-Head Attention (MHA) inside the low-rank bottleneck of the vision encoder and (ii) selectively applying weight decomposition only to the trainable low-rank branch rather than the full adapted weight. This design enables temporally-aware updates while preserving a frozen backbone and stable scaling. By mixing information across frames within the adaptation subspace, TemporalDoRA steers updates toward temporally consistent visual cues and improves robustness with minimal parameter overhead. To benchmark this setting, we present REAL-Colon-VQA, a colonoscopy VideoQA dataset with 6,424 clip--question pairs, including paired rephrased Out-of-Template questions to evaluate sensitivity to linguistic variation. TemporalDoRA improves Out-of-Template performance, and ablation studies confirm that temporal mixing inside the low-rank branch is the primary driver of these gains. We also validate on EndoVis18-VQA adapted to short clips and observe consistent improvements on the Out-of-Template split. Code and dataset available at~hrefhttps://anonymous.4open.science/r/TemporalDoRA-BFC8/Anonymous GitHub.