# IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment

arXiv: https://arxiv.org/abs/2303.09097 · HTML: https://mlsanity.com/paper/2303.09097/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 20 of 20)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study](https://mlsanity.com/paper/2002.00772/) (2020)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Action Quality Assessment Across Multiple Actions](https://mlsanity.com/paper/1812.06367/) (2018)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [Learning to score the figure skating sports videos](https://mlsanity.com/paper/1802.02774/) (2018)
- [The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos](https://mlsanity.com/paper/1812.05538/) (2018)
- [Am I a Baller? Basketball Performance Assessment from First-Person Videos](https://mlsanity.com/paper/1611.05365/) (2016)
- [Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment](https://mlsanity.com/paper/1901.07694/) (2019)
- [Manipulation-skill Assessment from Videos with Spatial Attention Network](https://mlsanity.com/paper/1901.02579/) (2019)
- [Towards Relatable Explainable AI with the Perceptual Process](https://mlsanity.com/paper/2112.14005/) (2021)
- [Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation](https://mlsanity.com/paper/2101.06030/) (2021)
- [Automatic Generation of Natural Language Explanations](https://mlsanity.com/paper/1707.01561/) (2017)
- [IF-City: Intelligible Fair City Planning to Measure, Explain and Mitigate Inequality](https://mlsanity.com/paper/2202.07349/) (2022)

## Built on by (most cited first, 9 of 9)

- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [RICA2: Rubric-Informed, Calibrated Assessment of Actions](https://mlsanity.com/paper/2408.02138/) (2024)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [TechCoach: Towards Technical-Point-Aware Descriptive Action Coaching](https://mlsanity.com/paper/2411.17130/) (2024)
- [MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment](https://mlsanity.com/paper/2605.08945/) (2026)
- [SketchXplain: Intuitive Visual Explanations of Image Classifiers with Sketches](https://mlsanity.com/paper/2606.17646/) (2026)
- [Latent Visual Diffusion Reasoning with Monte Carlo Tree Search](https://mlsanity.com/paper/2606.27988/) (2026)
- [AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations](https://mlsanity.com/paper/2608.14130/) (2026)
- [MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching](https://mlsanity.com/paper/2608.26094/) (2026)

## Abstract (arXiv)

AI-driven Action Quality Assessment (AQA) of sports videos can mimic Olympic judges to help score performances as a second opinion or for training. However, these AI methods are uninterpretable and do not justify their scores, which is important for algorithmic accountability. Indeed, to account for their decisions, instead of scoring subjectively, sports judges use a consistent set of criteria - rubric - on multiple actions in each performance sequence. Therefore, we propose IRIS to perform Interpretable Rubric-Informed Segmentation on action sequences for AQA. We investigated IRIS for scoring videos of figure skating performance. IRIS predicts (1) action segments, (2) technical element score differences of each segment relative to base scores, (3) multiple program component scores, and (4) the summed final score. In a modeling study, we found that IRIS performs better than non-interpretable, state-of-the-art models. In a formative user study, practicing figure skaters agreed with the rubric-informed explanations, found them useful, and trusted AI judgments more. This work highlights the importance of using judgment rubrics to account for AI decisions.