# Uncertainty-Driven Action Quality Assessment

arXiv: https://arxiv.org/abs/2207.14513 · HTML: https://mlsanity.com/paper/2207.14513/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders](https://mlsanity.com/paper/2004.05763/) (2020)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos](https://mlsanity.com/paper/1812.05538/) (2018)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [Uncertainty Inspired RGB-D Saliency Detection](https://mlsanity.com/paper/2009.03075/) (2020)
- [Action Quality Assessment with Temporal Parsing Transformer](https://mlsanity.com/paper/2207.09270/) (2022)
- [Manipulation-skill Assessment from Videos with Spatial Attention Network](https://mlsanity.com/paper/1901.02579/) (2019)
- [Conditional Variational Image Deraining](https://mlsanity.com/paper/2004.11373/) (2020)
- [Piano Skills Assessment](https://mlsanity.com/paper/2101.04884/) (2021)
- [RevPHiSeg: A Memory-Efficient Neural Network for Uncertainty Quantification in Medical Image Segmentation](https://mlsanity.com/paper/2008.06999/) (2020)

## Built on by (most cited first, 7 of 7)

- [RICA2: Rubric-Informed, Calibrated Assessment of Actions](https://mlsanity.com/paper/2408.02138/) (2024)
- [Are you Struggling? Dataset and Baselines for Struggle Determination in Assembly Videos](https://mlsanity.com/paper/2402.11057/) (2024)
- [ZEAL: Surgical Skill Assessment with Zero-shot Tool Inference Using Unified Foundation Model](https://mlsanity.com/paper/2407.02738/) (2024)
- [Towards Open-World Co-Salient Object Detection with Generative Uncertainty-aware Group Selective Exchange-Masking](https://mlsanity.com/paper/2310.10264/) (2023)
- [REST-HANDS: Rehabilitation with Egocentric Vision Using Smartglasses for Treatment of Hands after Surviving Stroke](https://mlsanity.com/paper/2409.20116/) (2024)
- [Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation](https://mlsanity.com/paper/2608.25736/) (2026)
- [MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching](https://mlsanity.com/paper/2608.26094/) (2026)

## Abstract (arXiv)

Automatic action quality assessment (AQA) has attracted increasing attention due to its wide applications. However, most existing AQA methods employ deterministic models to predict the final score for each action, while overlooking the subjectivity and diversity among expert judges during the scoring process. In this paper, we propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to utilize and capture the diversity among multiple judge scores. Specifically, we design a Conditional Variational Auto-Encoder (CVAE)-based module to encode the uncertainty in expert assessment, where multiple judge scores can be produced by sampling latent features from the learned latent space multiple times. To further utilize the uncertainty, we generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss, effectively reducing the influence of uncertain samples during training. Moreover, we further design an uncertainty-guided training strategy to dynamically adjust the learning order of the samples from low uncertainty to high uncertainty. The experiments show that our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.