# The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos

arXiv: https://arxiv.org/abs/1812.05538 · HTML: https://mlsanity.com/paper/1812.05538/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2018.

## Built on by (most cited first, 30 of 69)

- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Towards Unified Surgical Skill Assessment](https://mlsanity.com/paper/2106.01035/) (2021)
- [TSA-Net: Tube Self-Attention Network for Action Quality Assessment](https://mlsanity.com/paper/2201.03746/) (2022)
- [EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding](https://mlsanity.com/paper/2406.08877/) (2024)
- [Action Quality Assessment with Temporal Parsing Transformer](https://mlsanity.com/paper/2207.09270/) (2022)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment](https://mlsanity.com/paper/2202.14019/) (2022)
- [Test-time Fourier Style Calibration for Domain Generalization](https://mlsanity.com/paper/2205.06427/) (2022)
- [Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression](https://mlsanity.com/paper/2501.03674/) (2025)
- [Multimodal Action Quality Assessment](https://mlsanity.com/paper/2402.09444/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [Narrative Action Evaluation with Prompt-Guided Multimodal Interaction](https://mlsanity.com/paper/2404.14471/) (2024)
- [Auto-Encoding Score Distribution Regression for Action Quality Assessment](https://mlsanity.com/paper/2111.11029/) (2021)
- [BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimation](https://mlsanity.com/paper/2503.20781/) (2025)
- [Piano Skills Assessment](https://mlsanity.com/paper/2101.04884/) (2021)
- [A Survey of Video-based Action Quality Assessment](https://mlsanity.com/paper/2204.09271/) (2022)
- [Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessment](https://mlsanity.com/paper/2403.13798/) (2024)
- [Video Action Differencing](https://mlsanity.com/paper/2503.07860/) (2025)
- [Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling](https://mlsanity.com/paper/2309.17105/) (2023)
- [KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation](https://mlsanity.com/paper/2105.04181/) (2021)
- [Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos](https://mlsanity.com/paper/2110.05697/) (2021)
- [IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment](https://mlsanity.com/paper/2303.09097/) (2023)
- [Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment](https://mlsanity.com/paper/2304.14623/) (2023)
- [ExAct: A Video-Language Benchmark for Expert Action Analysis](https://mlsanity.com/paper/2506.06277/) (2025)

## Abstract (arXiv)

We present a new model to determine relative skill from long videos, through learnable temporal attention modules. Skill determination is formulated as a ranking problem, making it suitable for common and generic tasks. However, for long videos, parts of the video are irrelevant for assessing skill, and there may be variability in the skill exhibited throughout a video. We therefore propose a method which assesses the relative overall level of skill in a long video by attending to its skill-relevant parts. Our approach trains temporal attention modules, learned with only video-level supervision, using a novel rank-aware loss function. In addition to attending to task relevant video parts, our proposed loss jointly trains two attention modules to separately attend to video parts which are indicative of higher (pros) and lower (cons) skill. We evaluate our approach on the EPIC-Skills dataset and additionally annotate a larger dataset from YouTube videos for skill determination with five previously unexplored tasks. Our method outperforms previous approaches and classic softmax attention on both datasets by over 4% pairwise accuracy, and as much as 12% on individual tasks. We also demonstrate our model's ability to attend to rank-aware parts of the video.