# TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research

arXiv: https://arxiv.org/abs/2511.07412 · HTML: https://mlsanity.com/paper/2511.07412/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 18 of 18)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [FoundationStereo: Zero-Shot Stereo Matching](https://mlsanity.com/paper/2501.09898/) (2025)
- [ViTPose++: Vision Transformer for Generic Body Pose Estimation](https://mlsanity.com/paper/2212.04246/) (2022)
- [SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos](https://mlsanity.com/paper/2112.13715/) (2021)
- [Open-Sourced Reinforcement Learning Environments for Surgical Robotics](https://mlsanity.com/paper/1903.02090/) (2019)
- [ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling](https://mlsanity.com/paper/2404.07031/) (2024)
- [Online Reasoning Video Segmentation with Just-in-Time Digital Twins](https://mlsanity.com/paper/2503.21056/) (2025)
- [Creating a Digital Twin of Spinal Surgery: A Proof of Concept](https://mlsanity.com/paper/2403.16736/) (2024)
- [FluoroSAM: A Language-promptable Foundation Model for Flexible X-ray Image Segmentation](https://mlsanity.com/paper/2403.08059/) (2024)
- [Virtual Reality for Synergistic Surgical Training and Data Generation](https://mlsanity.com/paper/2111.08097/) (2021)
- [StraightTrack: Towards Mixed Reality Navigation System for Percutaneous K-wire Insertion](https://mlsanity.com/paper/2410.01143/) (2024)
- [Privacy-Preserving Operating Room Workflow Analysis using Digital Twins](https://mlsanity.com/paper/2504.12552/) (2025)
- [Comparative Evaluation of RGB-D SLAM Methods for Humanoid Robot Localization and Mapping](https://mlsanity.com/paper/2401.02816/) (2024)
- [Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin Representation](https://mlsanity.com/paper/2410.20026/) (2024)
- [dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scale](https://mlsanity.com/paper/2503.05646/) (2025)
- [Egosurg: Arbitrary view synthesis for egocentric replay of operating room workflows from ambient cameras](https://mlsanity.com/paper/2510.04802/) (2025)

## Built on by (most cited first, 1 of 1)

- [GEAR: GEometry-motion Alternating Refinement for Articulated Object Modeling with Gaussian Splatting](https://mlsanity.com/paper/2604.07728/) (2026)

## Abstract (arXiv)

Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, safety regulations and operational constraints in operating rooms (ORs) limit agents from freely perceiving and interacting in realistic settings. Digital twins provide high-fidelity, risk-free environments for exploration and training. How we may create dynamic digital representations of ORs that capture relevant spatial, visual, and behavioral complexity remains an open challenge. We introduce TwinOR, a real-to-sim infrastructure for constructing photorealistic and dynamic digital twins of ORs. The system reconstructs static geometry and continuously models human and equipment motion. The static and dynamic components are fused into an immersive 3D environment that supports controllable simulation and facilitates future embodied exploration. The proposed framework reconstructs complete OR geometry with centimeter-level accuracy while preserving dynamic interaction across surgical workflows. In our experiments, TwinOR synthesizes stereo and monocular RGB streams as well as depth observations for geometry understanding and visual localization tasks. Models such as FoundationStereo and ORB-SLAM3 evaluated on TwinOR-synthesized data achieve performance within their reported accuracy ranges on real-world indoor datasets, demonstrating that TwinOR provides sensor-level realism sufficient for emulating real-world perception and localization challenge. By establishing a perception-grounded real-to-sim pipeline, TwinOR enables the automatic construction of dynamic, photorealistic digital twins of ORs. As a safe and scalable environment for experimentation, TwinOR opens new opportunities for translating embodied intelligence from simulation to real-world clinical environments.