# Group-aware Contrastive Regression for Action Quality Assessment

arXiv: https://arxiv.org/abs/2108.07797 · HTML: https://mlsanity.com/paper/2108.07797/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Learning to score the figure skating sports videos](https://mlsanity.com/paper/1802.02774/) (2018)
- [The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos](https://mlsanity.com/paper/1812.05538/) (2018)
- [Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors](https://mlsanity.com/paper/1505.04868/) (2015)
- [Am I a Baller? Basketball Performance Assessment from First-Person Videos](https://mlsanity.com/paper/1611.05365/) (2016)
- [Video and Accelerometer-Based Motion Analysis for Automated Surgical Skills Assessment](https://mlsanity.com/paper/1702.07772/) (2017)
- [Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks](https://mlsanity.com/paper/1608.08128/) (2016)
- [Who's Better? Who's Best? Pairwise Deep Ranking for Skill Determination](https://mlsanity.com/paper/1703.09913/) (2017)

## Built on by (most cited first, 30 of 75)

- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo](https://mlsanity.com/paper/2203.17248/) (2022)
- [Action Quality Assessment with Temporal Parsing Transformer](https://mlsanity.com/paper/2207.09270/) (2022)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [GAIA: Rethinking Action Quality Assessment for AI-Generated Videos](https://mlsanity.com/paper/2406.06087/) (2024)
- [Hierarchical Classification Auxiliary Network for Time Series Forecasting](https://mlsanity.com/paper/2405.18975/) (2024)
- [Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment](https://mlsanity.com/paper/2202.14019/) (2022)
- [FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment](https://mlsanity.com/paper/2405.06887/) (2024)
- [Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision](https://mlsanity.com/paper/2407.06189/) (2024)
- [Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression](https://mlsanity.com/paper/2501.03674/) (2025)
- [Multimodal Action Quality Assessment](https://mlsanity.com/paper/2402.09444/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessment](https://mlsanity.com/paper/2311.07603/) (2023)
- [Narrative Action Evaluation with Prompt-Guided Multimodal Interaction](https://mlsanity.com/paper/2404.14471/) (2024)
- [Auto-Encoding Score Distribution Regression for Action Quality Assessment](https://mlsanity.com/paper/2111.11029/) (2021)
- [ExpertAF: Expert Actionable Feedback from Video](https://mlsanity.com/paper/2408.00672/) (2024)
- [BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimation](https://mlsanity.com/paper/2503.20781/) (2025)
- [PPGF: Probability Pattern-Guided Time Series Forecasting](https://mlsanity.com/paper/2502.12802/) (2025)
- [Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessment](https://mlsanity.com/paper/2403.13798/) (2024)
- [RICA2: Rubric-Informed, Calibrated Assessment of Actions](https://mlsanity.com/paper/2408.02138/) (2024)
- [Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling](https://mlsanity.com/paper/2309.17105/) (2023)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [PSCR: Patches Sampling-based Contrastive Regression for AIGC Image Quality Assessment](https://mlsanity.com/paper/2312.05897/) (2023)
- [PHI: Bridging Domain Shift in Long-Term Action Quality Assessment via Progressive Hierarchical Instruction](https://mlsanity.com/paper/2505.19972/) (2025)
- [Learning Sparse Temporal Video Mapping for Action Quality Assessment in Floor Gymnastics](https://mlsanity.com/paper/2301.06103/) (2023)
- [IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment](https://mlsanity.com/paper/2303.09097/) (2023)
- [Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metric](https://mlsanity.com/paper/2411.16619/) (2024)

## Abstract (arXiv)

Assessing action quality is challenging due to the subtle differences between videos and large variations in scores. Most existing approaches tackle this problem by regressing a quality score from a single video, suffering a lot from the large inter-video score variations. In this paper, we show that the relations among videos can provide important clues for more accurate action quality assessment during both training and inference. Specifically, we reformulate the problem of action quality assessment as regressing the relative scores with reference to another video that has shared attributes (e.g., category and difficulty), instead of learning unreferenced scores. Following this formulation, we propose a new Contrastive Regression (CoRe) framework to learn the relative scores by pair-wise comparison, which highlights the differences between videos and guides the models to learn the key hints for assessment. In order to further exploit the relative information between two videos, we devise a group-aware regression tree to convert the conventional score regression into two easier sub-problems: coarse-to-fine classification and regression in small intervals. To demonstrate the effectiveness of CoRe, we conduct extensive experiments on three mainstream AQA datasets including AQA-7, MTL-AQA and JIGSAWS. Our approach outperforms previous methods by a large margin and establishes new state-of-the-art on all three benchmarks.