# CTRN: Class-Temporal Relational Network for Action Detection

arXiv: https://arxiv.org/abs/2110.13473 · HTML: https://mlsanity.com/paper/2110.13473/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [Temporal Gaussian Mixture Layer for Videos](https://mlsanity.com/paper/1803.06316/) (2018)
- [Modeling Multi-Label Action Dependencies for Temporal Action Localization](https://mlsanity.com/paper/2103.03027/) (2021)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Learning Latent Super-Events to Detect Multiple Activities in Videos](https://mlsanity.com/paper/1712.01938/) (2017)
- [Asynchronous Temporal Fields for Action Recognition](https://mlsanity.com/paper/1612.06371/) (2016)
- [TAN: Temporal Aggregation Network for Dense Multi-label Action Recognition](https://mlsanity.com/paper/1812.06203/) (2018)

## Built on by (most cited first, 15 of 15)

- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [Token Turing Machines](https://mlsanity.com/paper/2211.09119/) (2022)
- [PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points](https://mlsanity.com/paper/2210.11035/) (2022)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [UniMD: Towards Unifying Moment Retrieval and Temporal Action Detection](https://mlsanity.com/paper/2404.04933/) (2024)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [JOADAA: joint online action detection and action anticipation](https://mlsanity.com/paper/2309.06130/) (2023)
- [PAT: Position-Aware Transformer for Dense Multi-Label Action Detection](https://mlsanity.com/paper/2308.05051/) (2023)
- [THORN: Temporal Human-Object Relation Network for Action Recognition](https://mlsanity.com/paper/2204.09468/) (2022)
- [MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videos](https://mlsanity.com/paper/2501.06138/) (2025)
- [Co-Occurrence Matters: Learning Action Relation for Temporal Action Localization](https://mlsanity.com/paper/2303.08463/) (2023)
- [AAN: Attributes-Aware Network for Temporal Action Detection](https://mlsanity.com/paper/2309.00696/) (2023)
- [An Effective-Efficient Approach for Dense Multi-Label Action Detection](https://mlsanity.com/paper/2406.06187/) (2024)
- [Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy](https://mlsanity.com/paper/2501.18509/) (2025)

## Abstract (arXiv)

Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. There are many real-world challenges in those datasets, such as composite action, co-occurring action, and high temporal variation of instance duration. For handling these challenges, we propose to explore both the class and temporal relations of detected actions. In this work, we introduce an end-to-end network: Class-Temporal Relational Network (CTRN). It contains three key components: (1) The Representation Transform Module filters the class-specific features from the mixed representations to build graph-structured data. (2) The Class-Temporal Module models the class and temporal relations in a sequential manner. (3) G-classifier leverages the privileged knowledge of the snippet-wise co-occurring action pairs to further improve the co-occurring action detection. We evaluate CTRN on three challenging densely labelled datasets and achieve state-of-the-art performance, reflecting the effectiveness and robustness of our method.