# Know your sensORs -- A Modality Study For Surgical Action Classification

arXiv: https://arxiv.org/abs/2203.08674 · HTML: https://mlsanity.com/paper/2203.08674/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 74 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.06811/) (2020)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation](https://mlsanity.com/paper/1808.08180/) (2018)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)
- [Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery](https://mlsanity.com/paper/2006.16166/) (2020)
- [PyTorchVideo: A Deep Learning Library for Video Understanding](https://mlsanity.com/paper/2111.09887/) (2021)
- [Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data](https://mlsanity.com/paper/2008.01034/) (2020)
- [A Robotic 3D Perception System for Operating Room Environment Awareness](https://mlsanity.com/paper/2003.09487/) (2020)
- [Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments](https://mlsanity.com/paper/2112.03750/) (2021)
- [Surgical Gesture Recognition with Optical Flow only](https://mlsanity.com/paper/1904.01143/) (2019)
- [Surgical Data Science: A Consensus Perspective](https://mlsanity.com/paper/1806.03184/) (2018)
- [Efficient Global Optimization of Non-differentiable, Symmetric Objectives for Multi Camera Placement](https://mlsanity.com/paper/2103.11210/) (2021)

## Built on by (most cited first, 6 of 6)

- [DisguisOR: Holistic Face Anonymization for the Operating Room](https://mlsanity.com/paper/2307.14241/) (2023)
- [Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room](https://mlsanity.com/paper/2503.13028/) (2025)
- [TrackOR: Towards Personalized Intelligent Operating Rooms Through Robust Tracking](https://mlsanity.com/paper/2508.07968/) (2025)
- [COMPOSE: Hypergraph Cover Optimization for Multi-view 3D Human Pose Estimation](https://mlsanity.com/paper/2601.09698/) (2026)
- [Mitigating Biases in Surgical Operating Rooms with Geometry](https://mlsanity.com/paper/2508.08028/) (2025)
- [OR-Action: Multi-Role Video Understanding with Fine-Grained Actions](https://mlsanity.com/paper/2606.13332/) (2026)

## Abstract (arXiv)

The surgical operating room (OR) presents many opportunities for automation and optimization. Videos from various sources in the OR are becoming increasingly available. The medical community seeks to leverage this wealth of data to develop automated methods to advance interventional care, lower costs, and improve overall patient outcomes. Existing datasets from OR room cameras are thus far limited in size or modalities acquired, leaving it unclear which sensor modalities are best suited for tasks such as recognizing surgical action from videos. This study demonstrates that surgical action recognition performance can vary depending on the image modalities used. We perform a methodical analysis on several commonly available sensor modalities, presenting two fusion approaches that improve classification performance. The analyses are carried out on a set of multi-view RGB-D video recordings of 18 laparoscopic procedures.