Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery
Not on the map. Why?
Abstract (arXiv)
As surgical AI transitions from pixel-level detection to complex reasoning, Scene Graphs (SGs) offer the structured, relational representations necessary to decode dynamic surgical environments. This PRISMA-ScR-guided scoping review systematically maps the evolving landscape of SG research in surgery, analyzing 52 primary studies to chart applications and methodological shifts. Our analysis reveals rapid growth, yet uncovers a critical 'data divide': internal-view research (e.g., triplet recognition from endoscopic video) accounts for 81% of studies and almost exclusively uses real-world 2D video, while external-view operating room modeling relies heavily on simulated data. Methodologically, we identify a decisive shift from foundational graph neural networks to specialized foundation models and generative AI, which together now account for approximately 50% of research in 2025. Crucially, our synthesis suggests that Scene Graphs are evolving from simple descriptors into essential 'neuro-symbolic guardrails', providing the structured, verifiable intermediate representation needed to prevent hallucinations in increasingly autonomous Surgical Foundation Models. Despite this promise, a major translational gap remains: none of the reviewed studies have proceeded to prospective clinical validation. We conclude that bridging this gap requires moving beyond standard computer vision metrics; we therefore propose the 'Validation Trinity' -- prioritizing Semantic Query Success, Latency-Aware Accuracy, and Safety-Critical Recall -- as the necessary evaluation framework to bring graph-based surgical AI into clinical practice.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 43 of its 95 references that are in the map. First posted on arXiv 25 Sep 2025.
Builds on
43 of 95 references.
References · 43
- 2017Attention Is All You Needcited by 60,361
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 367
- 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 141
- 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
- 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
- 2020Semantic Image Manipulation Using Scene Graphscited by 82
- 2022Scene Graph Generation: A Comprehensive Surveycited by 73
- 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
- 2020Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Tripletscited by 66
- 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
- 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
31 more
- 2018MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimationcited by 49
- 20224D-OR: Semantic Scene Graphs for OR Domain Modelingcited by 49
- 2022CholecTriplet2021: A benchmark challenge for surgical action triplet recognitioncited by 44
- 2023Advancing Surgical VQA with Scene Graph Knowledgecited by 41
- 2022Latent Graph Representations for Critical View of Safety Assessmentcited by 37
- 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
- 2020Learning and Reasoning with the Graph Structure Representation in Robotic Surgerycited by 29
- 2022Global-Reasoned Multi-Task Learning Model for Surgical Scene Understandingcited by 29
- 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
- 2022Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognitioncited by 27
- 2023CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detectioncited by 23
- 2023Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactionscited by 23
- 2023LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Roomscited by 21
- 2023Dynamic Scene Graph Representation for Surgical Videocited by 21
- 2024EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videoscited by 20
- 2023Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generationcited by 14
- 2024ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modelingcited by 14
- 2022Concept Graph Neural Networks for Surgical Video Understandingcited by 13
- 2024SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Predictioncited by 12
- 2024VISAGE: Video Synthesis using Action Graphs for Surgerycited by 11
- 2024Instrument-tissue Interaction Detection Framework for Surgical Video Understandingcited by 8
- 2021Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedurescited by 6
- 2024Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Roomscited by 6
- 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learningcited by 5
- 2019Compact Scene Graphs for Layout Composition and Patch Retrievalcited by 4
- 2022Task-Aware Asynchronous Multi-Task Model with Class Incremental Contrastive Learning for Surgical Scene Understandingcited by 4
- 2024Voxel Scene Graph for Intracranial Hemorrhagecited by 4
- 2025SurGrID: Controllable Surgical Simulation via Scene Graph to Image Diffusioncited by 3
- 2025Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglementcited by 3
- 2025Specialized Foundation Models for Intelligent Operating Roomscited by 3
- 2025Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgerycited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
- 2025Recognizing Surgical Phases Anywhere: Few-Shot Test-time Adaptation and Task-graph Guided Refinement
- 2026SurgGraph: Quantitative Laparoscopic Video Understanding via Geometry-Grounded Scene Graphs ↗
- 2026Surgical Video Generation From Diffusion to World Models: A Survey
- 2026A vision-language model and platform for temporally mapping surgery from video
- 2026A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling
- 2025SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis
- 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery