# Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks

arXiv: https://arxiv.org/abs/2109.11192 · HTML: https://mlsanity.com/paper/2109.11192/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 64 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021)
- [Exploiting the potential of unlabeled endoscopic video data with self-supervised learning](https://mlsanity.com/paper/1711.09726/) (2017)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [Recognizing Surgical Activities with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06329/) (2016)
- [Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks](https://mlsanity.com/paper/1908.07319/) (2019)
- [Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction](https://mlsanity.com/paper/2003.04772/) (2020)
- [SATR-DL: Improving Surgical Skill Assessment and Task Recognition in Robot-assisted Surgery with Deep Neural Networks](https://mlsanity.com/paper/1806.05798/) (2018)
- [Data-driven Holistic Framework for Automated Laparoscope Optimal View Control with Learning-based Depth Perception](https://mlsanity.com/paper/2011.11241/) (2020)

## Built on by (most cited first, 1 of 1)

- [Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation](https://mlsanity.com/paper/2412.06454/) (2024)

## Abstract (arXiv)

Robotic-assisted surgeries benefit both surgeons and patients, however, surgeons frequently need to adjust the endoscopic camera to achieve good viewpoints. Simultaneously controlling the camera and the surgical instruments is impossible, and consequentially, these camera adjustments repeatedly interrupt the surgery. Autonomous camera control could help overcome this challenge, but most existing systems are reactive, e.g., by having the camera follow the surgical instruments. We propose a predictive approach for anticipating when camera movements will occur using artificial neural networks. We used the kinematic data of the surgical instruments, which were recorded during robotic-assisted surgical training on porcine models. We split the data into segments, and labeled each either as a segment that immediately precedes a camera movement, or one that does not. Due to the large class imbalance, we trained an ensemble of networks, each on a balanced sub-set of the training data. We found that the instruments' kinematic data can be used to predict when camera movements will occur, and evaluated the performance on different segment durations and ensemble sizes. We also studied how much in advance an upcoming camera movement can be predicted, and found that predicting a camera movement 0.25, 0.5, and 1 second before they occurred achieved 98%, 94%, and 84% accuracy relative to the prediction of an imminent camera movement. This indicates that camera movement events can be predicted early enough to leave time for computing and executing an autonomous camera movement and suggests that an autonomous camera controller for RAMIS may one day be feasible.