# Creating a Digital Twin of Spinal Surgery: A Proof of Concept

arXiv: https://arxiv.org/abs/2403.16736 · HTML: https://mlsanity.com/paper/2403.16736/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 25 of 25)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction](https://mlsanity.com/paper/2212.05231/) (2022)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Synthesizing Diverse Human Motions in 3D Indoor Scenes](https://mlsanity.com/paper/2305.12411/) (2023)
- [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation](https://mlsanity.com/paper/1808.08180/) (2018)
- [Advancing Surgical VQA with Scene Graph Knowledge](https://mlsanity.com/paper/2312.10251/) (2023)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [You Only Learn Once: Universal Anatomical Landmark Detection](https://mlsanity.com/paper/2103.04657/) (2021)
- [ResFields: Residual Neural Fields for Spatiotemporal Signals](https://mlsanity.com/paper/2309.03160/) (2023)
- [LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms](https://mlsanity.com/paper/2303.13293/) (2023)
- [Dynamic Scene Graph Representation for Surgical Video](https://mlsanity.com/paper/2309.14538/) (2023)
- [A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms](https://mlsanity.com/paper/1701.07372/) (2017)
- [Robotic Surgery With Lean Reinforcement Learning](https://mlsanity.com/paper/2105.01006/) (2021)
- [Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery](https://mlsanity.com/paper/2308.02917/) (2023)
- [Dynamic Depth-Supervised NeRF for Multi-View RGB-D Operating Room Images](https://mlsanity.com/paper/2211.12436/) (2022)
- [Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures](https://mlsanity.com/paper/2106.15309/) (2021)
- [HMD-EgoPose: Head-Mounted Display-Based Egocentric Marker-Less Tool and Hand Pose Estimation for Augmented Surgical Guidance](https://mlsanity.com/paper/2202.11891/) (2022)
- [Articulated Clinician Detection Using 3D Pictorial Structures on RGB-D Data](https://mlsanity.com/paper/1602.03468/) (2016)
- [Dense 3D Reconstruction Through Lidar: A Comparative Study on Ex-vivo Porcine Tissue](https://mlsanity.com/paper/2401.10709/) (2024)

## Built on by (most cited first, 9 of 9)

- [SplatFormer: Point Transformer for Robust 3D Gaussian Splatting](https://mlsanity.com/paper/2411.06390/) (2024)
- [Automatic Calibration of a Multi-Camera System with Limited Overlapping Fields of View for 3D Surgical Scene Reconstruction](https://mlsanity.com/paper/2501.16221/) (2025)
- [Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin Representation](https://mlsanity.com/paper/2410.20026/) (2024)
- [Acquiring Submillimeter-Accurate Multi-Task Vision Datasets for Computer-Assisted Orthopedic Surgery](https://mlsanity.com/paper/2501.15371/) (2025)
- [TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research](https://mlsanity.com/paper/2511.07412/) (2025)
- [RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systems](https://mlsanity.com/paper/2511.14948/) (2025)
- [Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes](https://mlsanity.com/paper/2510.24332/) (2025)
- [Three-Dimensional Anatomical Data Generation Based on Artificial Neural Networks](https://mlsanity.com/paper/2511.19198/) (2025)
- [Training-free Detection and 6D Pose Estimation of Unseen Surgical Instruments](https://mlsanity.com/paper/2603.25228/) (2026)

## Abstract (arXiv)

Surgery digitalization is the process of creating a virtual replica of real-world surgery, also referred to as a surgical digital twin (SDT). It has significant applications in various fields such as education and training, surgical planning, and automation of surgical tasks. In addition, SDTs are an ideal foundation for machine learning methods, enabling the automatic generation of training data. In this paper, we present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. The proposed digitalization focuses on the acquisition and modelling of the geometry and appearance of the entire surgical scene. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion. We justify the proposed methodology, discuss the challenges faced and further extensions of our prototype. While our PoC partially relies on manual data curation, its high quality and great potential motivate the development of automated methods for the creation of SDTs.