# A Robotic 3D Perception System for Operating Room Environment Awareness

arXiv: https://arxiv.org/abs/2003.09487 · HTML: https://mlsanity.com/paper/2003.09487/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2020.

## Built on by (most cited first, 13 of 13)

- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [Temporally Consistent Online Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2111.09337/) (2021)
- [Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilities](https://mlsanity.com/paper/2308.03193/) (2023)
- [Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room](https://mlsanity.com/paper/2108.11801/) (2021)
- [SurgMAE: Masked Autoencoders for Long Surgical Video Analysis](https://mlsanity.com/paper/2305.11451/) (2023)
- [Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures](https://mlsanity.com/paper/2106.15309/) (2021)
- [Know your sensORs -- A Modality Study For Surgical Action Classification](https://mlsanity.com/paper/2203.08674/) (2022)
- [Multi-Modal Unsupervised Pre-Training for Surgical Operating Room Workflow Analysis](https://mlsanity.com/paper/2207.07894/) (2022)
- [ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room](https://mlsanity.com/paper/2312.12250/) (2023)
- [Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room](https://mlsanity.com/paper/2503.13028/) (2025)
- [Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Action](https://mlsanity.com/paper/2606.15647/) (2026)
- [Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness](https://mlsanity.com/paper/2407.05448/) (2024)
- [Self-Supervised Uncalibrated Multi-View Video Anonymization in the Operating Room](https://mlsanity.com/paper/2602.02850/) (2026)

## Abstract (arXiv)

Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentation algorithm is created to enable recognition of common and salient objects/equipment and surgical activities in a da Vinci OR. Our proposed 3D semantic segmentation method has been trained and validated on a novel densely annotated dataset that has been captured from clinical scenarios. Results: The results show that our proposed architecture has acceptable registration error (3.3%±1.4% of object-camera distance) and can robustly improve scene segmentation performance (mean Intersection Over Union - mIOU) for less frequently appearing classes (≥ 0.013) compared to a single-view method. Conclusion: We present the first dynamic multi-view perception system with a novel segmentation architecture, which can be used as a building block technology for applications such as surgical workflow analysis, automation of surgical sub-tasks and advanced guidance systems.