# Towards Holistic Surgical Scene Graph

arXiv: https://arxiv.org/abs/2507.15541 · HTML: https://mlsanity.com/paper/2507.15541/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 10 of 10)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [Towards Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2212.04582/) (2022)
- [Learning and Reasoning with the Graph Structure Representation in Robotic Surgery](https://mlsanity.com/paper/2007.03357/) (2020)
- [Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions](https://mlsanity.com/paper/2307.09548/) (2023)
- [Dynamic Scene Graph Representation for Surgical Video](https://mlsanity.com/paper/2309.14538/) (2023)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)

## Built on by (most cited first, 2 of 2)

- [SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?](https://mlsanity.com/paper/2605.01911/) (2026)
- [SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition](https://mlsanity.com/paper/2608.02188/) (2026)

## Abstract (arXiv)

Surgical scene understanding is crucial for computer-assisted intervention systems, requiring visual comprehension of surgical scenes that involves diverse elements such as surgical tools, anatomical structures, and their interactions. To effectively represent the complex information in surgical scenes, graph-based approaches have been explored to structurally model surgical entities and their relationships. Previous surgical scene graph studies have demonstrated the feasibility of representing surgical scenes using graphs. However, certain aspects of surgical scenes-such as diverse combinations of tool-action-target and the identity of the hand operating the tool-remain underexplored in graph-based representations, despite their importance. To incorporate these aspects into graph representations, we propose Endoscapes-SG201 dataset, which includes annotations for tool-action-target combinations and hand identity. We also introduce SSG-Com, a graph-based method designed to learn and represent these critical elements. Through experiments on downstream tasks such as critical view of safety assessment and action triplet recognition, we demonstrated the importance of integrating these essential scene graph components, highlighting their significant contribution to surgical scene understanding. The code and dataset are available at https://github.com/ailab-kyunghee/SSG-Com