# Surgical Skill Assessment via Video Semantic Aggregation

arXiv: https://arxiv.org/abs/2208.02611 · HTML: https://mlsanity.com/paper/2208.02611/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 4 Aug 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [ActionVLAD: Learning spatio-temporal aggregation for action classification](https://mlsanity.com/paper/1704.02895/) (2017)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [Towards Unified Surgical Skill Assessment](https://mlsanity.com/paper/2106.01035/) (2021)
- [Manipulation-skill Assessment from Videos with Spatial Attention Network](https://mlsanity.com/paper/1901.02579/) (2019)
- [Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessment](https://mlsanity.com/paper/1702.07772/) (2017)
- [Automated Surgical Skill Assessment in RMIS Training](https://mlsanity.com/paper/1712.08604/) (2017)
- [Evaluating surgical skills from kinematic data using convolutional neural networks](https://mlsanity.com/paper/1806.02750/) (2018)
- [Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery](https://mlsanity.com/paper/1806.05796/) (2018)
- [Spatio-Temporal Perturbations for Video Attribution](https://mlsanity.com/paper/2109.00222/) (2021)

## Built on by (most cited first, 13 of 13)

- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessment](https://mlsanity.com/paper/2311.07603/) (2023)
- [Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling](https://mlsanity.com/paper/2309.17105/) (2023)
- [Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis](https://mlsanity.com/paper/2510.16371/) (2025)
- [Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment](https://mlsanity.com/paper/2509.09327/) (2025)
- [ZEAL: Surgical Skill Assessment with Zero-shot Tool Inference Using Unified Foundation Model](https://mlsanity.com/paper/2407.02738/) (2024)
- [ReCAP: Recursive Cross Attention Network for Pseudo-Label Generation in Robotic Surgical Skill Assessment](https://mlsanity.com/paper/2407.05180/) (2024)
- [AI-Driven Evaluation of Surgical Skill via Action Recognition](https://mlsanity.com/paper/2512.24411/) (2025)
- [CoRe-DA: Contrastive Regression for Unsupervised Domain Adaptation in Surgical Skill Assessment](https://mlsanity.com/paper/2603.29666/) (2026)
- [SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessment](https://mlsanity.com/paper/2603.00108/) (2026)
- [An Integrated Video-AI Platform for Action-Level Microanastomosis Training and Performance Feedback](https://mlsanity.com/paper/2609.06380/) (2026)
- [Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos](https://mlsanity.com/paper/2610.01205/) (2026)

## Abstract (arXiv)

Automated video-based assessment of surgical skills is a promising task in assisting young surgical trainees, especially in poor-resource areas. Existing works often resort to a CNN-LSTM joint framework that models long-term relationships by LSTMs on spatially pooled short-term CNN features. However, this practice would inevitably neglect the difference among semantic concepts such as tools, tissues, and background in the spatial dimension, impeding the subsequent temporal relationship modeling. In this paper, we propose a novel skill assessment framework, Video Semantic Aggregation (ViSA), which discovers different semantic parts and aggregates them across spatiotemporal dimensions. The explicit discovery of semantic parts provides an explanatory visualization that helps understand the neural network's decisions. It also enables us to further incorporate auxiliary information such as the kinematic data to improve representation learning and performance. The experiments on two datasets show the competitiveness of ViSA compared to state-of-the-art methods. Source code is available at: bit.ly/MICCAI2022ViSA.