# Enriching Local and Global Contexts for Temporal Action Localization

arXiv: https://arxiv.org/abs/2107.12960 · HTML: https://mlsanity.com/paper/2107.12960/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles](https://mlsanity.com/paper/1811.09795/) (2018)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals](https://mlsanity.com/paper/1703.06189/) (2017)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [Fast Learning of Temporal Action Proposal via Dense Boundary Generator](https://mlsanity.com/paper/1911.04127/) (2019)
- [Boundary Content Graph Neural Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/2008.01432/) (2020)
- [Multi-granularity Generator for Temporal Action Proposal](https://mlsanity.com/paper/1811.11524/) (2018)
- [Learning Spatio-Temporal Representation with Local and Global Diffusion](https://mlsanity.com/paper/1906.05571/) (2019)
- [Two-Stream Consensus Network for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2010.11594/) (2020)
- [CTAP: Complementary Temporal Action Proposal Generation](https://mlsanity.com/paper/1807.04821/) (2018)
- [Temporal Context Network for Activity Localization in Videos](https://mlsanity.com/paper/1708.02349/) (2017)
- [A robust and efficient video representation for action recognition](https://mlsanity.com/paper/1504.05524/) (2015)
- [ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2103.15088/) (2021)

## Built on by (most cited first, 30 of 64)

- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [UnLoc: A Unified Framework for Video Localization Tasks](https://mlsanity.com/paper/2308.11062/) (2023)
- [ReAct: Temporal Action Detection with Relational Queries](https://mlsanity.com/paper/2207.07097/) (2022)
- [An Empirical Study of End-to-End Temporal Action Detection](https://mlsanity.com/paper/2204.02932/) (2022)
- [Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2203.16800/) (2022)
- [WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition](https://mlsanity.com/paper/2304.05088/) (2023)
- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [Proposal-Free Temporal Action Detection via Global Segmentation Mask Learning](https://mlsanity.com/paper/2207.06580/) (2022)
- [TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localization](https://mlsanity.com/paper/2303.09055/) (2023)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [GateHUB: Gated History Unit with Background Suppression for Online Action Detection](https://mlsanity.com/paper/2206.04668/) (2022)
- [OpenTAL: Towards Open Set Temporal Action Localization](https://mlsanity.com/paper/2203.05114/) (2022)
- [PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points](https://mlsanity.com/paper/2210.11035/) (2022)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Colar: Effective and Efficient Online Action Detection by Consulting Exemplars](https://mlsanity.com/paper/2203.01057/) (2022)
- [Proposal-Based Multiple Instance Learning for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2305.17861/) (2023)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [Multi-modal Prompting for Low-Shot Temporal Action Localization](https://mlsanity.com/paper/2303.11732/) (2023)
- [DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion](https://mlsanity.com/paper/2303.14863/) (2023)
- [DyFADet: Dynamic Feature Aggregation for Temporal Action Detection](https://mlsanity.com/paper/2407.03197/) (2024)
- [Learning to Refactor Action and Co-occurrence Features for Temporal Action Localization](https://mlsanity.com/paper/2206.11493/) (2022)
- [Improving Audio-Visual Video Parsing with Pseudo Visual Labels](https://mlsanity.com/paper/2303.02344/) (2023)
- [UniMD: Towards Unifying Moment Retrieval and Temporal Action Detection](https://mlsanity.com/paper/2404.04933/) (2024)
- [Temporal Action Localization with Enhanced Instant Discriminability](https://mlsanity.com/paper/2309.05590/) (2023)
- [Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2212.09335/) (2022)
- [Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection](https://mlsanity.com/paper/2303.17285/) (2023)
- [Action Detection via an Image Diffusion Process](https://mlsanity.com/paper/2404.01051/) (2024)
- [Faster-TAD: Towards Temporal Action Detection with Proposal Generation and Classification in a Unified Network](https://mlsanity.com/paper/2204.02674/) (2022)

## Abstract (arXiv)

Effectively tackling the problem of temporal action localization (TAL) necessitates a visual representation that jointly pursues two confounding goals, i.e., fine-grained discrimination for temporal localization and sufficient visual invariance for action classification. We address this challenge by enriching both the local and global contexts in the popular two-stage temporal localization framework, where action proposals are first generated followed by action classification and temporal boundary regression. Our proposed model, dubbed ContextLoc, can be divided into three sub-networks: L-Net, G-Net and P-Net. L-Net enriches the local context via fine-grained modeling of snippet-level features, which is formulated as a query-and-retrieval process. G-Net enriches the global context via higher-level modeling of the video-level representation. In addition, we introduce a novel context adaptation module to adapt the global context to different proposals. P-Net further models the context-aware inter-proposal relations. We explore two existing models to be the P-Net in our experiments. The efficacy of our proposed method is validated by experimental results on the THUMOS14 (54.3% at tIoU@0.5) and ActivityNet v1.3 (56.01% at tIoU@0.5) datasets, which outperforms recent states of the art. Code is available at https://github.com/buxiangzhiren/ContextLoc.