# Auto-Encoding Score Distribution Regression for Action Quality Assessment

arXiv: https://arxiv.org/abs/2111.11029 · HTML: https://mlsanity.com/paper/2111.11029/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Data Uncertainty Learning in Face Recognition](https://mlsanity.com/paper/2003.11339/) (2020)
- [Label Distribution Learning](https://mlsanity.com/paper/1408.6027/) (2014)
- [Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving](https://mlsanity.com/paper/1904.04620/) (2019)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos](https://mlsanity.com/paper/1812.05538/) (2018)
- [Hamiltonian Variational Auto-Encoder](https://mlsanity.com/paper/1805.11328/) (2018)
- [Am I a Baller? Basketball Performance Assessment from First-Person Videos](https://mlsanity.com/paper/1611.05365/) (2016)
- [Uncertainty Estimation in One-Stage Object Detection](https://mlsanity.com/paper/1905.10296/) (2019)
- [A Hierarchical Neural Autoencoder for Paragraphs and Documents](https://mlsanity.com/paper/1506.01057/) (2015)
- [C3AE: Exploring the Limits of Compact Model for Age Estimation](https://mlsanity.com/paper/1904.05059/) (2019)
- [Fine-Grained Age Estimation in the wild with Attention LSTM Networks](https://mlsanity.com/paper/1805.10445/) (2018)
- [Long Activity Video Understanding using Functional Object-Oriented Network](https://mlsanity.com/paper/1807.00983/) (2018)
- [Extraction and Classification of Diving Clips from Continuous Video Footage](https://mlsanity.com/paper/1705.09003/) (2017)

## Built on by (most cited first, 18 of 18)

- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [ASTRA: An Action Spotting TRAnsformer for Soccer Videos](https://mlsanity.com/paper/2404.01891/) (2024)
- [RICA2: Rubric-Informed, Calibrated Assessment of Actions](https://mlsanity.com/paper/2408.02138/) (2024)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Interpretable Long-term Action Quality Assessment](https://mlsanity.com/paper/2408.11687/) (2024)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [Are you Struggling? Dataset and Baselines for Struggle Determination in Assembly Videos](https://mlsanity.com/paper/2402.11057/) (2024)
- [Fine-grained Action Analysis: A Multi-modality and Multi-task Dataset of Figure Skating](https://mlsanity.com/paper/2307.02730/) (2023)
- [Action Quality Assessment using Transformers](https://mlsanity.com/paper/2207.12318/) (2022)
- [Unlabeled Action Quality Assessment Based on Multi-dimensional Adaptive Constrained Dynamic Time Warping](https://mlsanity.com/paper/2410.14161/) (2024)
- [An X3D Neural Network Analysis for Runner's Performance Assessment in a Wild Sporting Environment](https://mlsanity.com/paper/2307.12183/) (2023)
- [ZEAL: Surgical Skill Assessment with Zero-shot Tool Inference Using Unified Foundation Model](https://mlsanity.com/paper/2407.02738/) (2024)
- [REST-HANDS: Rehabilitation with Egocentric Vision Using Smartglasses for Treatment of Hands after Surviving Stroke](https://mlsanity.com/paper/2409.20116/) (2024)
- [Focus on What Matters: Constraining Spatial-Temporal Attention via Action-Units for Noise-Resilient AQA](https://mlsanity.com/paper/2511.05611/) (2025)
- [ProSkill: Segment-Level Skill Assessment in Procedural Videos](https://mlsanity.com/paper/2601.20661/) (2026)
- [Contrastive learning-based video quality assessment-jointed video vision transformer for video recognition](https://mlsanity.com/paper/2603.10965/) (2026)
- [Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment](https://mlsanity.com/paper/2607.07438/) (2026)
- [MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching](https://mlsanity.com/paper/2608.26094/) (2026)

## Abstract (arXiv)

The action quality assessment (AQA) of videos is a challenging vision task since the relation between videos and action scores is difficult to model. Thus, AQA has been widely studied in the literature. Traditionally, AQA is treated as a regression problem to learn the underlying mappings between videos and action scores. But previous methods ignored data uncertainty in AQA dataset. To address aleatoric uncertainty, we further develop a plug-and-play module Distribution Auto-Encoder (DAE). Specifically, it encodes videos into distributions and uses the reparameterization trick in variational auto-encoders (VAE) to sample scores, which establishes a more accurate mapping between videos and scores. Meanwhile, a likelihood loss is used to learn the uncertainty parameters. We plug our DAE approach into MUSDL and CoRe. Experimental results on public datasets demonstrate that our method achieves state-of-the-art on AQA-7, MTL-AQA, and JIGSAWS datasets. Our code is available at https://github.com/InfoX-SEU/DAE-AQA.