# LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

arXiv: https://arxiv.org/abs/2303.13293 · HTML: https://mlsanity.com/paper/2303.13293/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer](https://mlsanity.com/paper/2103.09712/) (2021)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy](https://mlsanity.com/paper/2011.08065/) (2020)
- [Target Adaptive Context Aggregation for Video Scene Graph Generation](https://mlsanity.com/paper/2108.08121/) (2021)
- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery](https://mlsanity.com/paper/2006.16166/) (2020)
- [Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition](https://mlsanity.com/paper/2211.16963/) (2022)
- [CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data](https://mlsanity.com/paper/2203.09475/) (2022)
- [Surgical Workflow Recognition: from Analysis of Challenges to Architectural Study](https://mlsanity.com/paper/2203.09230/) (2022)
- [Multi-Modal Unsupervised Pre-Training for Surgical Operating Room Workflow Analysis](https://mlsanity.com/paper/2207.07894/) (2022)
- [Adaptation of Surgical Activity Recognition Models Across Operating Rooms](https://mlsanity.com/paper/2207.03083/) (2022)

## Built on by (most cited first, 21 of 21)

- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding](https://mlsanity.com/paper/2412.18450/) (2024)
- [ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling](https://mlsanity.com/paper/2404.07031/) (2024)
- [Creating a Digital Twin of Spinal Surgery: A Proof of Concept](https://mlsanity.com/paper/2403.16736/) (2024)
- [Generalizable Operating Room Expert with Multimodal Enhancement](https://mlsanity.com/paper/2508.08199/) (2025)
- [Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms](https://mlsanity.com/paper/2404.09231/) (2024)
- [Voxel Scene Graph for Intracranial Hemorrhage](https://mlsanity.com/paper/2407.21580/) (2024)
- [Specialized Foundation Models for Intelligent Operating Rooms](https://mlsanity.com/paper/2505.12890/) (2025)
- [Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery](https://mlsanity.com/paper/2509.20941/) (2025)
- [Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Action](https://mlsanity.com/paper/2606.15647/) (2026)
- [S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR](https://mlsanity.com/paper/2402.14461/) (2024)
- [Optimizing Latent Graph Representations of Surgical Scenes for Zero-Shot Domain Transfer](https://mlsanity.com/paper/2403.06953/) (2024)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition](https://mlsanity.com/paper/2502.13883/) (2025)
- [Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes](https://mlsanity.com/paper/2510.24332/) (2025)
- [Self-Supervised Uncalibrated Multi-View Video Anonymization in the Operating Room](https://mlsanity.com/paper/2602.02850/) (2026)
- [Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video](https://mlsanity.com/paper/2603.17178/) (2026)
- [PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms](https://mlsanity.com/paper/2603.19920/) (2026)
- [Where are they looking in the operating room?](https://mlsanity.com/paper/2604.20574/) (2026)
- [NoPA: Non-Parametric Online 3D Scene Graph Generation](https://mlsanity.com/paper/2607.00529/) (2026)
- [SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms](https://mlsanity.com/paper/2608.15336/) (2026)

## Abstract (arXiv)

Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the operating room (OR) is, therefore, a challenging but essential task, with the potential to optimize the performance of surgical teams and aid in developing new surgical technologies to improve patient outcomes. The holistic representation of surgical scenes as semantic scene graphs (SGG), where entities are represented as nodes and relations between them as edges, is a promising direction for fine-grained semantic OR understanding. We propose, for the first time, the use of temporal information for more accurate and consistent holistic OR modeling. Specifically, we introduce memory scene graphs, where the scene graphs of previous time steps act as the temporal representation guiding the current prediction. We design an end-to-end architecture that intelligently fuses the temporal information of our lightweight memory scene graphs with the visual information from point clouds and images. We evaluate our method on the 4D-OR dataset and demonstrate that integrating temporality leads to more accurate and consistent results achieving an +5% increase and a new SOTA of 0.88 in macro F1. This work opens the path for representing the entire surgery history with memory scene graphs and improves the holistic understanding in the OR. Introducing scene graphs as memory representations can offer a valuable tool for many temporal understanding tasks.