# Video-based fully automatic assessment of open surgery suturing skills

arXiv: https://arxiv.org/abs/2110.13972 · HTML: https://mlsanity.com/paper/2110.13972/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021)
- [Automated Surgical Skill Assessment in RMIS Training](https://mlsanity.com/paper/1712.08604/) (2017)
- [Evaluating surgical skills from kinematic data using convolutional neural networks](https://mlsanity.com/paper/1806.02750/) (2018)
- [Optimizing the Trade-off between Single-Stage and Two-Stage Object Detectors using Image Difficulty Prediction](https://mlsanity.com/paper/1803.08707/) (2018)
- [Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery](https://mlsanity.com/paper/2012.06948/) (2020)
- [Machine Learning Approach for Skill Evaluation in Robotic-Assisted Surgery](https://mlsanity.com/paper/1611.05136/) (2016)

## Built on by (most cited first, 12 of 12)

- [Video-based Formative and Summative Assessment of Surgical Tasks using Deep Learning](https://mlsanity.com/paper/2203.09589/) (2022)
- [Using Hand Pose Estimation To Automate Open Surgery Training Feedback](https://mlsanity.com/paper/2211.07021/) (2022)
- [Bounded Future MS-TCN++ for surgical gesture recognition](https://mlsanity.com/paper/2209.14647/) (2022)
- [Open surgery tool classification and hand utilization using a multi-camera system](https://mlsanity.com/paper/2111.06098/) (2021)
- [Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth Camera](https://mlsanity.com/paper/2401.10037/) (2024)
- [MS-TCRNet: Multi-Stage Temporal Convolutional Recurrent Networks for Action Segmentation Using Sensor-Augmented Kinematics](https://mlsanity.com/paper/2303.07814/) (2023)
- [ZEAL: Surgical Skill Assessment with Zero-shot Tool Inference Using Unified Foundation Model](https://mlsanity.com/paper/2407.02738/) (2024)
- [AI-Driven Evaluation of Surgical Skill via Action Recognition](https://mlsanity.com/paper/2512.24411/) (2025)
- [SFGANS Self-supervised Future Generator for human ActioN Segmentation](https://mlsanity.com/paper/2401.00438/) (2023)
- [OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025](https://mlsanity.com/paper/2605.22200/) (2026)
- [ExpOS: Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction](https://mlsanity.com/paper/2605.23653/) (2026)
- [An Integrated Video-AI Platform for Action-Level Microanastomosis Training and Performance Feedback](https://mlsanity.com/paper/2609.06380/) (2026)

## Abstract (arXiv)

The goal of this study was to develop new reliable open surgery suturing simulation system for training medical students in situation where resources are limited or in the domestic setup. Namely, we developed an algorithm for tools and hands localization as well as identifying the interactions between them based on simple webcam video data, calculating motion metrics for assessment of surgical skill. Twenty-five participants performed multiple suturing tasks using our simulator. The YOLO network has been modified to a multi-task network, for the purpose of tool localization and tool-hand interaction detection. This was accomplished by splitting the YOLO detection heads so that they supported both tasks with minimal addition to computer run-time. Furthermore, based on the outcome of the system, motion metrics were calculated. These metrics included traditional metrics such as time and path length as well as new metrics assessing the technique participants use for holding the tools. The dual-task network performance was similar to that of two networks, while computational load was only slightly bigger than one network. In addition, the motion metrics showed significant differences between experts and novices. While video capture is an essential part of minimally invasive surgery, it is not an integral component of open surgery. Thus, new algorithms, focusing on the unique challenges open surgery videos present, are required. In this study, a dual-task network was developed to solve both a localization task and a hand-tool interaction task. The dual network may be easily expanded to a multi-task network, which may be useful for images with multiple layers and for evaluating the interaction between these different layers.