# Video-based Formative and Summative Assessment of Surgical Tasks using Deep Learning

arXiv: https://arxiv.org/abs/2203.09589 · HTML: https://mlsanity.com/paper/2203.09589/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://mlsanity.com/paper/1803.02579/) (2018)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [Deep Learning on Small Datasets without Pre-Training using Cosine Loss](https://mlsanity.com/paper/1901.09054/) (2019)
- [Automated Surgical Skill Assessment in RMIS Training](https://mlsanity.com/paper/1712.08604/) (2017)
- [Evaluating surgical skills from kinematic data using convolutional neural networks](https://mlsanity.com/paper/1806.02750/) (2018)
- [How Much Can We Really Trust You? Towards Simple, Interpretable Trust Quantification Metrics for Deep Neural Networks](https://mlsanity.com/paper/2009.05835/) (2020)
- [Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks](https://mlsanity.com/paper/1908.07319/) (2019)
- [Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery](https://mlsanity.com/paper/1806.05796/) (2018)
- [Video-based fully automatic assessment of open surgery suturing skills](https://mlsanity.com/paper/2110.13972/) (2021)
- [Deep Neural Networks for the Assessment of Surgical Skills: A Systematic Review](https://mlsanity.com/paper/2103.05113/) (2021)
- [Where Does Trust Break Down? A Quantitative Trust Analysis of Deep Neural Networks via Trust Matrix and Conditional Trust Densities](https://mlsanity.com/paper/2009.14701/) (2020)
- [Machine Learning Approach for Skill Evaluation in Robotic-Assisted Surgery](https://mlsanity.com/paper/1611.05136/) (2016)
- [Who's Better? Who's Best? Pairwise Deep Ranking for Skill Determination](https://mlsanity.com/paper/1703.09913/) (2017)

## Built on by (most cited first, 8 of 8)

- [One-shot skill assessment in high-stakes domains with limited data via meta learning](https://mlsanity.com/paper/2301.00812/) (2022)
- [Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication](https://mlsanity.com/paper/2412.16195/) (2024)
- [ZEAL: Surgical Skill Assessment with Zero-shot Tool Inference Using Unified Foundation Model](https://mlsanity.com/paper/2407.02738/) (2024)
- [ReCAP: Recursive Cross Attention Network for Pseudo-Label Generation in Robotic Surgical Skill Assessment](https://mlsanity.com/paper/2407.05180/) (2024)
- [End-to-End Deep Learning for Real-Time Neuroimaging-Based Assessment of Bimanual Motor Skills](https://mlsanity.com/paper/2504.03681/) (2025)
- [Cognitive-Motor Integration in Assessing Bimanual Motor Skills](https://mlsanity.com/paper/2404.10889/) (2024)
- [Deep Learning for Video-Based Assessment of Endotracheal Intubation Skills](https://mlsanity.com/paper/2404.11727/) (2024)
- [Video-based Surgical Skill Assessment Using Dynamics-and-Uncertainty-Aware Tree-based Gaussian Process Classifier](https://mlsanity.com/paper/2609.24619/) (2026)

## Abstract (arXiv)

To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated - none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA remains manually- and time-intensive and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds and low-stakes formative assessment to guide surgical skill acquisition. Formative assessment is generated using heatmaps of visual features that correlate with surgical performance. Hence, the DL model paves the way to the quantitative and reproducible evaluation of surgical tasks from videos with the potential for broad dissemination in surgical training, certification, and credentialing.