# Generalizable Operating Room Expert with Multimodal Enhancement

arXiv: https://arxiv.org/abs/2508.08199 · HTML: https://mlsanity.com/paper/2508.08199/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence](https://mlsanity.com/paper/2505.23747/) (2025)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction](https://mlsanity.com/paper/2505.20279/) (2025)
- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](https://mlsanity.com/paper/1612.00814/) (2016)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms](https://mlsanity.com/paper/2303.13293/) (2023)
- [ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling](https://mlsanity.com/paper/2404.07031/) (2024)
- [MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding](https://mlsanity.com/paper/2506.08356/) (2025)
- [Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms](https://mlsanity.com/paper/2404.09231/) (2024)

## Built on by (most cited first, 7 of 7)

- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2512.16561/) (2025)
- [World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning](https://mlsanity.com/paper/2604.26934/) (2026)
- [Attention in Space: Functional Roles of VLM Heads for Spatial Reasoning](https://mlsanity.com/paper/2603.20662/) (2026)
- [UniHM: Unified Dexterous Hand Manipulation with Vision Language Model](https://mlsanity.com/paper/2603.00732/) (2026)
- [HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Model](https://mlsanity.com/paper/2508.11350/) (2025)
- [Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Action](https://mlsanity.com/paper/2606.15647/) (2026)

## Abstract (arXiv)

Precise spatial modeling in the operating room (OR) is essential for intraoperative awareness, hazard avoidance, and surgical decision-making. Although existing approaches exploit multimodal data to learn spatial relationships, many depend on sensing modalities that are difficult to deploy in real clinical environments and remain limited in explicit 3D reasoning under constrained sensing conditions. Meanwhile, models trained primarily on readily available 2D data often fail to capture the fine-grained geometric and semantic structure of complex OR scenes. To address these limitations, we introduce OR-Expert, a large vision-language framework for 3D spatial reasoning with RGB-only inference. OR-Expert internally derives depth, panoptic segmentation, and point-cloud cues from RGB images and encodes them as structured spatial representations. Its Spatial-Enhanced Feature Fusion Block aligns these pseudo-modalities with RGB and textual features in a shared token space, enabling joint semantic, geometric, and language reasoning. The unified end-to-end MLLM therefore supports detailed spatial understanding without requiring external depth, segmentation, or point-cloud sensors, or additional expert annotations at inference time. Experiments on multiple operating-room benchmarks demonstrate that OR-Expert achieves state-of-the-art performance and generalizes effectively to unseen surgical scenes and downstream spatial reasoning tasks.