# Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding

arXiv: https://arxiv.org/abs/2201.11957 · HTML: https://mlsanity.com/paper/2201.11957/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2022.

## Built on by (most cited first, 29 of 29)

- [Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer](https://mlsanity.com/paper/2206.11053/) (2022)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2305.11692/) (2023)
- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation](https://mlsanity.com/paper/2308.07156/) (2023)
- [SAM Meets Robotic Surgery: An Empirical Study in Robustness Perspective](https://mlsanity.com/paper/2304.14674/) (2023)
- [SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation](https://mlsanity.com/paper/2408.04593/) (2024)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2307.05182/) (2023)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [Instrument-tissue Interaction Detection Framework for Surgical Video Understanding](https://mlsanity.com/paper/2404.00322/) (2024)
- [SegSTRONG-C: Segmenting Surgical Tools Robustly On Non-adversarial Generated Corruptions -- An EndoVis'24 Challenge](https://mlsanity.com/paper/2407.11906/) (2024)
- [Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Models](https://mlsanity.com/paper/2410.07753/) (2024)
- [Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registration](https://mlsanity.com/paper/2509.06467/) (2025)
- [SegMatch: A semi-supervised learning method for surgical instrument segmentation](https://mlsanity.com/paper/2308.05232/) (2023)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Task-Aware Asynchronous Multi-Task Model with Class Incremental Contrastive Learning for Surgical Scene Understanding](https://mlsanity.com/paper/2211.15327/) (2022)
- [ACT-Net: Anchor-context Action Detection in Surgery Videos](https://mlsanity.com/paper/2310.03377/) (2023)
- [Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery](https://mlsanity.com/paper/2509.20941/) (2025)
- [Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models](https://mlsanity.com/paper/2403.17192/) (2024)
- [EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery](https://mlsanity.com/paper/2506.06830/) (2025)
- [Surgical Video Understanding with Label Interpolation](https://mlsanity.com/paper/2509.18802/) (2025)
- [More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery](https://mlsanity.com/paper/2512.07596/) (2025)
- [Beyond Rigid AI: Towards Natural Human-Machine Symbiosis for Interoperative Surgical Assistance](https://mlsanity.com/paper/2507.23088/) (2025)
- [FASL-Seg: Anatomy and Tool Segmentation of Surgical Scenes](https://mlsanity.com/paper/2509.06159/) (2025)
- [Weakly Supervised Multicenter Nancy Index Scoring in Ulcerative Colitis Using Foundation Models](https://mlsanity.com/paper/2604.23706/) (2026)

## Abstract (arXiv)

Global and local relational reasoning enable scene understanding models to perform human-like scene analysis and understanding. Scene understanding enables better semantic segmentation and object-to-object interaction detection. In the medical domain, a robust surgical scene understanding model allows the automation of surgical skill evaluation, real-time monitoring of surgeon's performance and post-surgical analysis. This paper introduces a globally-reasoned multi-task surgical scene understanding model capable of performing instrument segmentation and tool-tissue interaction detection. Here, we incorporate global relational reasoning in the latent interaction space and introduce multi-scale local (neighborhood) reasoning in the coordinate space to improve segmentation. Utilizing the multi-task model setup, the performance of the visual-semantic graph attention network in interaction detection is further enhanced through global reasoning. The global interaction space features from the segmentation module are introduced into the graph network, allowing it to detect interactions based on both node-to-node and global interaction reasoning. Our model reduces the computation cost compared to running two independent single-task models by sharing common modules, which is indispensable for practical applications. Using a sequential optimization technique, the proposed multi-task model outperforms other state-of-the-art single-task models on the MICCAI endoscopic vision challenge 2018 dataset. Additionally, we also observe the performance of the multi-task model when trained using the knowledge distillation technique. The official code implementation is made available in GitHub.