# Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

arXiv: https://arxiv.org/abs/2007.03357 · HTML: https://mlsanity.com/paper/2007.03357/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 29 of 29)

- [Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer](https://mlsanity.com/paper/2206.11053/) (2022)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2305.11692/) (2023)
- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [Dynamic Scene Graph Representation for Surgical Video](https://mlsanity.com/paper/2309.14538/) (2023)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2109.05263/) (2021)
- [Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation](https://mlsanity.com/paper/2107.11091/) (2021)
- [CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2307.05182/) (2023)
- [Revisiting Distillation for Continual Learning on Visual Question Localized-Answering in Robotic Surgery](https://mlsanity.com/paper/2307.12045/) (2023)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding](https://mlsanity.com/paper/2505.24287/) (2025)
- [Instrument-tissue Interaction Detection Framework for Surgical Video Understanding](https://mlsanity.com/paper/2404.00322/) (2024)
- [Graph Convolutional Networks for Multi-modality Medical Imaging: Methods, Architectures, and Clinical Applications](https://mlsanity.com/paper/2202.08916/) (2022)
- [Towards Graph Representation Learning Based Surgical Workflow Anticipation](https://mlsanity.com/paper/2208.03824/) (2022)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Task-Aware Asynchronous Multi-Task Model with Class Incremental Contrastive Learning for Surgical Scene Understanding](https://mlsanity.com/paper/2211.15327/) (2022)
- [ACT-Net: Anchor-context Action Detection in Surgery Videos](https://mlsanity.com/paper/2310.03377/) (2023)
- [Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery](https://mlsanity.com/paper/2509.20941/) (2025)
- [Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis](https://mlsanity.com/paper/2306.16285/) (2023)
- [SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival](https://mlsanity.com/paper/2403.09290/) (2024)
- [Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation](https://mlsanity.com/paper/2412.06454/) (2024)
- [Towards Holistic Surgical Scene Graph](https://mlsanity.com/paper/2507.15541/) (2025)
- [Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding](https://mlsanity.com/paper/2212.11511/) (2022)
- [EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery](https://mlsanity.com/paper/2506.06830/) (2025)
- [TopoOR: A Unified Topological Scene Representation for the Operating Room](https://mlsanity.com/paper/2603.09466/) (2026)
- [Beyond Rigid AI: Towards Natural Human-Machine Symbiosis for Interoperative Surgical Assistance](https://mlsanity.com/paper/2507.23088/) (2025)
- [Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT](https://mlsanity.com/paper/2609.07857/) (2026)

## Abstract (arXiv)

Learning to infer graph representations and performing spatial reasoning in a complex surgical environment can play a vital role in surgical scene understanding in robotic surgery. For this purpose, we develop an approach to generate the scene graph and predict surgical interactions between instruments and surgical region of interest (ROI) during robot-assisted surgery. We design an attention link function and integrate with a graph parsing network to recognize the surgical interactions. To embed each node with corresponding neighbouring node features, we further incorporate SageConv into the network. The scene graph generation and active edge classification mostly depend on the embedding or feature extraction of node and edge features from complex image representation. Here, we empirically demonstrate the feature extraction methods by employing label smoothing weighted loss. Smoothing the hard label can avoid the over-confident prediction of the model and enhances the feature representation learned by the penultimate layer. To obtain the graph scene label, we annotate the bounding box and the instrument-ROI interactions on the robotic scene segmentation challenge 2018 dataset with an experienced clinical expert in robotic surgery and employ it to evaluate our propositions.