# Video-based surgical skill assessment using 3D convolutional neural networks

arXiv: https://arxiv.org/abs/1903.02306 · HTML: https://mlsanity.com/paper/1903.02306/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2019.

## Built on by (most cited first, 30 of 57)

- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [Towards Unified Surgical Skill Assessment](https://mlsanity.com/paper/2106.01035/) (2021)
- [Surgical Skill Assessment via Video Semantic Aggregation](https://mlsanity.com/paper/2208.02611/) (2022)
- [Video-based fully automatic assessment of open surgery suturing skills](https://mlsanity.com/paper/2110.13972/) (2021)
- [Deep Neural Networks for the Assessment of Surgical Skills: A Systematic Review](https://mlsanity.com/paper/2103.05113/) (2021)
- [Video-based Formative and Summative Assessment of Surgical Tasks using Deep Learning](https://mlsanity.com/paper/2203.09589/) (2022)
- [GynSurg: A Comprehensive Gynecology Laparoscopic Surgery Dataset](https://mlsanity.com/paper/2506.11356/) (2025)
- [Uncertainty-Driven Action Quality Assessment](https://mlsanity.com/paper/2207.14513/) (2022)
- [Interpretable Long-term Action Quality Assessment](https://mlsanity.com/paper/2408.11687/) (2024)
- [Towards a Medical AI Scientist](https://mlsanity.com/paper/2603.28589/) (2026)
- [Using Hand Pose Estimation To Automate Open Surgery Training Feedback](https://mlsanity.com/paper/2211.07021/) (2022)
- [Objective Surgical Skills Assessment and Tool Localization: Results from the MICCAI 2021 SimSurgSkill Challenge](https://mlsanity.com/paper/2212.04448/) (2022)
- [Bounded Future MS-TCN++ for surgical gesture recognition](https://mlsanity.com/paper/2209.14647/) (2022)
- [GLSFormer: Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos](https://mlsanity.com/paper/2307.11081/) (2023)
- [Event Recognition in Laparoscopic Gynecology Videos with Hybrid Transformers](https://mlsanity.com/paper/2312.00593/) (2023)
- [Improving Action Quality Assessment using Weighted Aggregation](https://mlsanity.com/paper/2102.10555/) (2021)
- [Runtime Detection of Executional Errors in Robot-Assisted Surgery](https://mlsanity.com/paper/2203.00737/) (2022)
- [One-shot skill assessment in high-stakes domains with limited data via meta learning](https://mlsanity.com/paper/2301.00812/) (2022)
- [Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos](https://mlsanity.com/paper/2406.19217/) (2024)
- [SkillSight: Efficient First-Person Skill Assessment with Gaze](https://mlsanity.com/paper/2511.19629/) (2025)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Uncertainty-aware Self-supervised Learning for Cross-domain Technical Skill Assessment in Robot-assisted Surgery](https://mlsanity.com/paper/2304.14589/) (2023)
- [CycleSAM: Few-Shot Surgical Scene Segmentation with Cycle- and Scene-Consistent Feature Matching](https://mlsanity.com/paper/2407.06795/) (2024)
- [Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis](https://mlsanity.com/paper/2510.16371/) (2025)
- [Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Camera](https://mlsanity.com/paper/2401.10037/) (2024)
- [Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication](https://mlsanity.com/paper/2412.16195/) (2024)
- [StepAL: Step-aware Active Learning for Cataract Surgical Videos](https://mlsanity.com/paper/2507.22059/) (2025)
- [Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment](https://mlsanity.com/paper/2509.09327/) (2025)
- [TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification](https://mlsanity.com/paper/2403.01802/) (2024)

## Abstract (arXiv)

Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Methods: Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a Temporal Segment Network during training. Results: We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1% to 100.0%. Conclusions: Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for manual feature engineering. Further evaluation will require more annotated data for training and testing.