# Manipulation-skill Assessment from Videos with Spatial Attention Network

arXiv: https://arxiv.org/abs/1901.02579 · HTML: https://mlsanity.com/paper/1901.02579/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 29 of 29)

- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Towards Unified Surgical Skill Assessment](https://mlsanity.com/paper/2106.01035/) (2021)
- [Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression](https://mlsanity.com/paper/2501.03674/) (2025)
- [Multimodal Action Quality Assessment](https://mlsanity.com/paper/2402.09444/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [Piano Skills Assessment](https://mlsanity.com/paper/2101.04884/) (2021)
- [A Survey of Video-based Action Quality Assessment](https://mlsanity.com/paper/2204.09271/) (2022)
- [Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessment](https://mlsanity.com/paper/2403.13798/) (2024)
- [Surgical Skill Assessment via Video Semantic Aggregation](https://mlsanity.com/paper/2208.02611/) (2022)
- [Deep Neural Networks for the Assessment of Surgical Skills: A Systematic Review](https://mlsanity.com/paper/2103.05113/) (2021)
- [IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment](https://mlsanity.com/paper/2303.09097/) (2023)
- [Uncertainty-Driven Action Quality Assessment](https://mlsanity.com/paper/2207.14513/) (2022)
- [Interpretable Long-term Action Quality Assessment](https://mlsanity.com/paper/2408.11687/) (2024)
- [Are you Struggling? Dataset and Baselines for Struggle Determination in Assembly Videos](https://mlsanity.com/paper/2402.11057/) (2024)
- [Win-Fail Action Recognition](https://mlsanity.com/paper/2102.07355/) (2021)
- [Leadership Assessment in Pediatric Intensive Care Unit Team Training](https://mlsanity.com/paper/2505.24389/) (2025)
- [Hand Hygiene Assessment via Joint Step Segmentation and Key Action Scorer](https://mlsanity.com/paper/2209.12221/) (2022)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)
- [A Spatial-Temporal Transformer based Framework For Human Pose Assessment And Correction in Education Scenarios](https://mlsanity.com/paper/2311.00401/) (2023)
- [Video-based Surgical Skill Assessment using Tree-based Gaussian Process Classifier](https://mlsanity.com/paper/2312.10208/) (2023)
- [Video-based Surgical Skill Assessment Using Dynamics-and-Uncertainty-Aware Tree-based Gaussian Process Classifier](https://mlsanity.com/paper/2609.24619/) (2026)

## Abstract (arXiv)

Recent advances in computer vision have made it possible to automatically assess from videos the manipulation skills of humans in performing a task, which breeds many important applications in domains such as health rehabilitation and manufacturing. Previous methods of video-based skill assessment did not consider the attention mechanism humans use in assessing videos, limiting their performance as only a small part of video regions is informative for skill assessment. Our motivation here is to estimate attention in videos that helps to focus on critically important video regions for better skill assessment. In particular, we propose a novel RNN-based spatial attention model that considers accumulated attention state from previous frames as well as high-level knowledge about the progress of an undergoing task. We evaluate our approach on a newly collected dataset of infant grasping task and four existing datasets of hand manipulation tasks. Experiment results demonstrate that state-of-the-art performance can be achieved by considering attention in automatic skill assessment.