# 3D Human Pose Estimation in Multi-View Operating Room Videos Using Differentiable Camera Projections

arXiv: https://arxiv.org/abs/2210.11826 · HTML: https://mlsanity.com/paper/2210.11826/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020)
- [MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation](https://mlsanity.com/paper/1808.08180/) (2018)
- [Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation](https://mlsanity.com/paper/1508.06708/) (2015)
- [A generalizable approach for multi-view 3D human pose regression](https://mlsanity.com/paper/1804.10462/) (2018)
- [A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms](https://mlsanity.com/paper/1701.07372/) (2017)

## Built on by (most cited first, 7 of 7)

- [Rendering Humans from Object-Occluded Monocular Videos](https://mlsanity.com/paper/2308.04622/) (2023)
- [Dynamic Depth-Supervised NeRF for Multi-View RGB-D Operating Room Images](https://mlsanity.com/paper/2211.12436/) (2022)
- [Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room](https://mlsanity.com/paper/2503.13028/) (2025)
- [Sports Analysis and VR Viewing System Based on Player Tracking and Pose Estimation with Multimodal and Multiview Sensors](https://mlsanity.com/paper/2405.01112/) (2024)
- [LiCamPose: Combining Multi-View LiDAR and RGB Cameras for Robust Single-timestamp 3D Human Pose Estimation](https://mlsanity.com/paper/2312.06409/) (2023)
- [mmHSense: Multi-Modal and Distributed mmWave ISAC Datasets for Human Sensing](https://mlsanity.com/paper/2509.21396/) (2025)
- [Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training](https://mlsanity.com/paper/2606.28104/) (2026)

## Abstract (arXiv)

3D human pose estimation in multi-view operating room (OR) videos is a relevant asset for person tracking and action recognition. However, the surgical environment makes it challenging to find poses due to sterile clothing, frequent occlusions, and limited public data. Methods specifically designed for the OR are generally based on the fusion of detected poses in multiple camera views. Typically, a 2D pose estimator such as a convolutional neural network (CNN) detects joint locations. Then, the detected joint locations are projected to 3D and fused over all camera views. However, accurate detection in 2D does not guarantee accurate localisation in 3D space. In this work, we propose to directly optimise for localisation in 3D by training 2D CNNs end-to-end based on a 3D loss that is backpropagated through each camera's projection parameters. Using videos from the MVOR dataset, we show that this end-to-end approach outperforms optimisation in 2D space.