# Temporal Relational Modeling with Self-Supervision for Action Segmentation

arXiv: https://arxiv.org/abs/2012.07508 · HTML: https://mlsanity.com/paper/2012.07508/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 29 of 29)

- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [How Much Temporal Long-Term Context is Needed for Action Segmentation?](https://mlsanity.com/paper/2308.11358/) (2023)
- [C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentation](https://mlsanity.com/paper/2212.11078/) (2022)
- [Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition](https://mlsanity.com/paper/2503.18595/) (2025)
- [HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding](https://mlsanity.com/paper/2307.05721/) (2023)
- [SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation](https://mlsanity.com/paper/2105.14158/) (2021)
- [Continuous Human Action Recognition for Human-Machine Interaction: A Review](https://mlsanity.com/paper/2202.13096/) (2022)
- [Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding](https://mlsanity.com/paper/2403.11463/) (2024)
- [Efficient Temporal Action Segmentation via Boundary-aware Query Voting](https://mlsanity.com/paper/2405.15995/) (2024)
- [Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos](https://mlsanity.com/paper/2209.05653/) (2022)
- [SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition](https://mlsanity.com/paper/2306.08859/) (2023)
- [Cross-Skeleton Interaction Graph Aggregation Network for Representation Learning of Mouse Social Behaviour](https://mlsanity.com/paper/2208.03819/) (2022)
- [Prompt-enhanced Hierarchical Transformer Elevating Cardiopulmonary Resuscitation Instruction via Temporal Action Segmentation](https://mlsanity.com/paper/2308.16552/) (2023)
- [Temporal Segment Transformer for Action Segmentation](https://mlsanity.com/paper/2302.13074/) (2023)
- [TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering](https://mlsanity.com/paper/2303.05166/) (2023)
- [DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation](https://mlsanity.com/paper/2304.02110/) (2023)
- [Text-Derived Relational Graph-Enhanced Network for Skeleton-Based Action Segmentation](https://mlsanity.com/paper/2503.15126/) (2025)
- [The Object at Hand: Automated Editing for Mixed Reality Video Guidance from Hand-Object Interactions](https://mlsanity.com/paper/2109.14744/) (2021)
- [CASR: Refining Action Segmentation via Marginalizing Frame-levle Causal Relationships](https://mlsanity.com/paper/2311.12401/) (2023)
- [SMC-NCA: Semantic-guided Multi-level Contrast for Semi-supervised Temporal Action Segmentation](https://mlsanity.com/paper/2312.12347/) (2023)
- [Improving action segmentation via explicit similarity measurement](https://mlsanity.com/paper/2502.10713/) (2025)
- [Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioning](https://mlsanity.com/paper/2605.31115/) (2026)
- [Cross Your Body: A Cognitive Assessment System for Children](https://mlsanity.com/paper/2111.12824/) (2021)
- [SFGANS Self-supervised Future Generator for human ActioN Segmentation](https://mlsanity.com/paper/2401.00438/) (2023)
- [LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation](https://mlsanity.com/paper/2603.24097/) (2026)
- [Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation](https://mlsanity.com/paper/2603.24134/) (2026)

## Abstract (arXiv)

Temporal relational modeling in video is essential for human action understanding, such as action recognition and action segmentation. Although Graph Convolution Networks (GCNs) have shown promising advantages in relation reasoning on many tasks, it is still a challenge to apply graph convolution networks on long video sequences effectively. The main reason is that large number of nodes (i.e., video frames) makes GCNs hard to capture and model temporal relations in videos. To tackle this problem, in this paper, we introduce an effective GCN module, Dilated Temporal Graph Reasoning Module (DTGRM), designed to model temporal relations and dependencies between video frames at various time spans. In particular, we capture and model temporal relations via constructing multi-level dilated temporal graphs where the nodes represent frames from different moments in video. Moreover, to enhance temporal reasoning ability of the proposed model, an auxiliary self-supervised task is proposed to encourage the dilated temporal graph reasoning module to find and correct wrong temporal relations in videos. Our DTGRM model outperforms state-of-the-art action segmentation models on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset. The code is available at https://github.com/redwang/DTGRM.