# Intra- and Inter-Action Understanding via Temporal Action Parsing

arXiv: https://arxiv.org/abs/2005.10229 · HTML: https://mlsanity.com/paper/2005.10229/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 30 of 41)

- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [FineAction: A Fine-Grained Video Dataset for Temporal Action Localization](https://mlsanity.com/paper/2105.11107/) (2021)
- [Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos](https://mlsanity.com/paper/2203.14104/) (2022)
- [Action Quality Assessment with Temporal Parsing Transformer](https://mlsanity.com/paper/2207.09270/) (2022)
- [FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs](https://mlsanity.com/paper/2407.02157/) (2024)
- [Exploring Segment-level Semantics for Online Phase Recognition from Surgical Videos](https://mlsanity.com/paper/2111.11044/) (2021)
- [Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression](https://mlsanity.com/paper/2501.03674/) (2025)
- [UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2111.14799/) (2021)
- [Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection](https://mlsanity.com/paper/2112.04771/) (2021)
- [NVIDIA Nemotron Nano V2 VL](https://mlsanity.com/paper/2511.03929/) (2025)
- [Learning Asynchronous and Sparse Human-Object Interaction in Videos](https://mlsanity.com/paper/2103.02758/) (2021)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection](https://mlsanity.com/paper/2203.00307/) (2022)
- [Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization](https://mlsanity.com/paper/2303.11003/) (2023)
- [Structured Context Transformer for Generic Event Boundary Detection](https://mlsanity.com/paper/2206.02985/) (2022)
- [FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance](https://mlsanity.com/paper/2505.13437/) (2025)
- [CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation](https://mlsanity.com/paper/2109.15170/) (2021)
- [Therbligs in Action: Video Understanding through Motion Primitives](https://mlsanity.com/paper/2304.03631/) (2023)
- [Rethinking the Architecture Design for Efficient Generic Event Boundary Detection](https://mlsanity.com/paper/2407.12622/) (2024)
- [Discerning Generic Event Boundaries in Long-Form Wild Videos](https://mlsanity.com/paper/2106.10090/) (2021)
- [Exploring Temporal Granularity in Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2112.04480/) (2021)
- [LEAP: LLM-Generation of Egocentric Action Programs](https://mlsanity.com/paper/2312.00055/) (2023)
- [Online Generic Event Boundary Detection](https://mlsanity.com/paper/2510.06855/) (2025)
- [What's in the Flow? Exploiting Temporal Motion Cues for Unsupervised Generic Event Boundary Detection](https://mlsanity.com/paper/2404.18935/) (2024)
- [Fine-grained Dynamic Network for Generic Event Boundary Detection](https://mlsanity.com/paper/2407.04274/) (2024)
- [LocoMotion: Learning Motion-Focused Video-Language Representations](https://mlsanity.com/paper/2410.12018/) (2024)
- [Motion Aware Self-Supervision for Generic Event Boundary Detection](https://mlsanity.com/paper/2210.05574/) (2022)
- [Fine-grained Action Analysis: A Multi-modality and Multi-task Dataset of Figure Skating](https://mlsanity.com/paper/2307.02730/) (2023)

## Abstract (arXiv)

Current methods for action recognition primarily rely on deep convolutional networks to derive feature embeddings of visual and motion features. While these methods have demonstrated remarkable performance on standard benchmarks, we are still in need of a better understanding as to how the videos, in particular their internal structures, relate to high-level semantics, which may lead to benefits in multiple aspects, e.g. interpretable predictions and even new methods that can take the recognition performances to a next level. Towards this goal, we construct TAPOS, a new dataset developed on sport videos with manual annotations of sub-actions, and conduct a study on temporal action parsing on top. Our study shows that a sport activity usually consists of multiple sub-actions and that the awareness of such temporal structures is beneficial to action recognition. We also investigate a number of temporal parsing methods, and thereon devise an improved method that is capable of mining sub-actions from training data without knowing the labels of them. On the constructed TAPOS, the proposed method is shown to reveal intra-action information, i.e. how action instances are made of sub-actions, and inter-action information, i.e. one specific sub-action may commonly appear in various actions.