# Dynamic Depth-Supervised NeRF for Multi-View RGB-D Operating Room Images

arXiv: https://arxiv.org/abs/2211.12436 · HTML: https://mlsanity.com/paper/2211.12436/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 12 of 12)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo](https://mlsanity.com/paper/2109.01129/) (2021)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [3D Human Pose Estimation in Multi-View Operating Room Videos Using Differentiable Camera Projections](https://mlsanity.com/paper/2210.11826/) (2022)

## Built on by (most cited first, 7 of 7)

- [Self-Evolving Depth-Supervised 3D Gaussian Splatting from Rendered Stereo Pairs](https://mlsanity.com/paper/2409.07456/) (2024)
- [Creating a Digital Twin of Spinal Surgery: A Proof of Concept](https://mlsanity.com/paper/2403.16736/) (2024)
- [RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos](https://mlsanity.com/paper/2412.03077/) (2024)
- [TeleOR: Real-time Telemedicine System for Full-Scene Operating Room](https://mlsanity.com/paper/2407.19763/) (2024)
- [Dynamic Gaussian Splatting from Markerless Motion Capture can Reconstruct Infants Movements](https://mlsanity.com/paper/2310.19441/) (2023)
- [Knowledge NeRF: Few-shot Novel View Synthesis for Dynamic Articulated Objects](https://mlsanity.com/paper/2404.00674/) (2024)
- [Geometry OR Tracker: Universal Geometric Operating Room Tracking](https://mlsanity.com/paper/2603.00560/) (2026)

## Abstract (arXiv)

The operating room (OR) is an environment of interest for the development of sensing systems, enabling the detection of people, objects, and their semantic relations. Due to frequent occlusions in the OR, these systems often rely on input from multiple cameras. While increasing the number of cameras generally increases algorithm performance, there are hard limitations to the number and locations of cameras in the OR. Neural Radiance Fields (NeRF) can be used to render synthetic views from arbitrary camera positions, virtually enlarging the number of cameras in the dataset. In this work, we explore the use of NeRF for view synthesis of dynamic scenes in the OR, and we show that regularisation with depth supervision from RGB-D sensor data results in higher image quality. We optimise a dynamic depth-supervised NeRF with up to six synchronised cameras that capture the surgical field in five distinct phases before and during a knee replacement surgery. We qualitatively inspect views rendered by a virtual camera that moves 180 degrees around the surgical field at differing time values. Quantitatively, we evaluate view synthesis from an unseen camera position in terms of PSNR, SSIM and LPIPS for the colour channels and in MAE and error percentage for the estimated depth. We find that NeRFs can be used to generate geometrically consistent views, also from interpolated camera positions and at interpolated time intervals. Views are generated from an unseen camera pose with an average PSNR of 18.2 and a depth estimation error of 2.0%. Our results show the potential of a dynamic NeRF for view synthesis in the OR and stress the relevance of depth supervision in a clinical setting.