# Interpretable Long-term Action Quality Assessment

arXiv: https://arxiv.org/abs/2408.11687 · HTML: https://mlsanity.com/paper/2408.11687/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [Learning to score the figure skating sports videos](https://mlsanity.com/paper/1802.02774/) (2018)
- [The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos](https://mlsanity.com/paper/1812.05538/) (2018)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [Video-based surgical skill assessment using 3D convolutional neural networks](https://mlsanity.com/paper/1903.02306/) (2019)
- [TSA-Net: Tube Self-Attention Network for Action Quality Assessment](https://mlsanity.com/paper/2201.03746/) (2022)
- [Action Quality Assessment with Temporal Parsing Transformer](https://mlsanity.com/paper/2207.09270/) (2022)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [Manipulation-skill Assessment from Videos with Spatial Attention Network](https://mlsanity.com/paper/1901.02579/) (2019)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Auto-Encoding Score Distribution Regression for Action Quality Assessment](https://mlsanity.com/paper/2111.11029/) (2021)
- [Improving Action Quality Assessment using Weighted Aggregation](https://mlsanity.com/paper/2102.10555/) (2021)

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [Trajectory-guided Motion Perception for Facial Expression Quality Assessment in Neurological Disorders](https://mlsanity.com/paper/2504.09530/) (2025)
- [MotionHalluc: Diagnosing Kinematic Hallucinations in Fine-Grained Motion Reasoning](https://mlsanity.com/paper/2606.23061/) (2026)
- [Latent Visual Diffusion Reasoning with Monte Carlo Tree Search](https://mlsanity.com/paper/2606.27988/) (2026)
- [Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment](https://mlsanity.com/paper/2607.07438/) (2026)
- [Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation](https://mlsanity.com/paper/2608.25736/) (2026)

## Abstract (arXiv)

Long-term Action Quality Assessment (AQA) evaluates the execution of activities in videos. However, the length presents challenges in fine-grained interpretability, with current AQA methods typically producing a single score by averaging clip features, lacking detailed semantic meanings of individual clips. Long-term videos pose additional difficulty due to the complexity and diversity of actions, exacerbating interpretability challenges. While query-based transformer networks offer promising long-term modeling capabilities, their interpretability in AQA remains unsatisfactory due to a phenomenon we term Temporal Skipping, where the model skips self-attention layers to prevent output degradation. To address this, we propose an attention loss function and a query initialization method to enhance performance and interpretability. Additionally, we introduce a weight-score regression module designed to approximate the scoring patterns observed in human judgments and replace conventional single-score regression, improving the rationality of interpretability. Our approach achieves state-of-the-art results on three real-world, long-term AQA benchmarks. Our code is available at: https://github.com/dx199771/Interpretability-AQA