# Co-Occurrence Matters: Learning Action Relation for Temporal Action Localization

arXiv: https://arxiv.org/abs/2303.08463 · HTML: https://mlsanity.com/paper/2303.08463/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection](https://mlsanity.com/paper/2103.01903/) (2021)
- [Multi-granularity Generator for Temporal Action Proposal](https://mlsanity.com/paper/1811.11524/) (2018)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [Dont Even Look Once: Synthesizing Features for Zero-Shot Detection](https://mlsanity.com/paper/1911.07933/) (2019)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition](https://mlsanity.com/paper/2111.01024/) (2021)
- [Temporal Gaussian Mixture Layer for Videos](https://mlsanity.com/paper/1803.06316/) (2018)
- [Modeling Multi-Label Action Dependencies for Temporal Action Localization](https://mlsanity.com/paper/2103.03027/) (2021)
- [Background Learnable Cascade for Zero-Shot Object Detection](https://mlsanity.com/paper/2010.04502/) (2020)
- [Learning Latent Super-Events to Detect Multiple Activities in Videos](https://mlsanity.com/paper/1712.01938/) (2017)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [CTRN: Class-Temporal Relational Network for Action Detection](https://mlsanity.com/paper/2110.13473/) (2021)

## Built on by (most cited first, 1 of 1)

- [MGCA-Net: Multi-Grained Category-Aware Network for Open-Vocabulary Temporal Action Localization](https://mlsanity.com/paper/2511.13039/) (2025)

## Abstract (arXiv)

Temporal action localization (TAL) is a prevailing task due to its great application potential. Existing works in this field mainly suffer from two weaknesses: (1) They often neglect the multi-label case and only focus on temporal modeling. (2) They ignore the semantic information in class labels and only use the visual information. To solve these problems, we propose a novel Co-Occurrence Relation Module (CORM) that explicitly models the co-occurrence relationship between actions. Besides the visual information, it further utilizes the semantic embeddings of class labels to model the co-occurrence relationship. The CORM works in a plug-and-play manner and can be easily incorporated with the existing sequence models. By considering both visual and semantic co-occurrence, our method achieves high multi-label relationship modeling capacity. Meanwhile, existing datasets in TAL always focus on low-semantic atomic actions. Thus we construct a challenging multi-label dataset UCF-Crime-TAL that focuses on high-semantic actions by annotating the UCF-Crime dataset at frame level and considering the semantic overlap of different events. Extensive experiments on two commonly used TAL datasets, textiti.e., MultiTHUMOS and TSU, and our newly proposed UCF-Crime-TAL demenstrate the effectiveness of the proposed CORM, which achieves state-of-the-art performance on these datasets.