# Proposal-Free Temporal Action Detection via Global Segmentation Mask Learning

arXiv: https://arxiv.org/abs/2207.06580 · HTML: https://mlsanity.com/paper/2207.06580/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 57 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [Disentangled Non-Local Neural Networks](https://mlsanity.com/paper/2006.06668/) (2020)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals](https://mlsanity.com/paper/1703.06189/) (2017)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [Video Self-Stitching Graph Network for Temporal Action Localization](https://mlsanity.com/paper/2011.14598/) (2020)

## Built on by (most cited first, 29 of 29)

- [Zero-Shot Temporal Action Detection via Vision-Language Prompting](https://mlsanity.com/paper/2207.08184/) (2022)
- [UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2308.14395/) (2023)
- [WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition](https://mlsanity.com/paper/2304.05088/) (2023)
- [TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localization](https://mlsanity.com/paper/2303.09055/) (2023)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion](https://mlsanity.com/paper/2303.14863/) (2023)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [HR-Pro: Point-supervised Temporal Action Localization via Hierarchical Reliability Propagation](https://mlsanity.com/paper/2308.12608/) (2023)
- [Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection](https://mlsanity.com/paper/2303.17285/) (2023)
- [Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions](https://mlsanity.com/paper/2403.20254/) (2024)
- [Action Detection via an Image Diffusion Process](https://mlsanity.com/paper/2404.01051/) (2024)
- [Temporal Action Localization for Inertial-based Human Activity Recognition](https://mlsanity.com/paper/2311.15831/) (2023)
- [Post-Processing Temporal Action Detection](https://mlsanity.com/paper/2211.14924/) (2022)
- [Prediction-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2408.16729/) (2024)
- [Temporal Action Detection Model Compression by Progressive Block Drop](https://mlsanity.com/paper/2503.16916/) (2025)
- [DiffSED: Sound Event Detection with Denoising Diffusion](https://mlsanity.com/paper/2308.07293/) (2023)
- [Zero-shot Action Localization via the Confidence of Large Vision-Language Models](https://mlsanity.com/paper/2410.14340/) (2024)
- [Multi-Modal Few-Shot Temporal Action Detection](https://mlsanity.com/paper/2211.14905/) (2022)
- [Boundary Discretization and Reliable Classification Network for Temporal Action Detection](https://mlsanity.com/paper/2310.06403/) (2023)
- [A Hybrid Graph Network for Complex Activity Detection in Video](https://mlsanity.com/paper/2310.17493/) (2023)
- [ContextDet: Temporal Action Detection with Adaptive Context Aggregation](https://mlsanity.com/paper/2410.15279/) (2024)
- [Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization](https://mlsanity.com/paper/2310.03456/) (2023)
- [DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding](https://mlsanity.com/paper/2506.23196/) (2025)
- [FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision](https://mlsanity.com/paper/2608.01392/) (2026)
- [Boosting Point-supervised Temporal Action Localization via Text Refinement and Alignment](https://mlsanity.com/paper/2602.01257/) (2026)
- [Masked Diffusion Vision-Language Models for Temporal Action Localization](https://mlsanity.com/paper/2605.29858/) (2026)
- [ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation](https://mlsanity.com/paper/2606.23835/) (2026)
- [MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization](https://mlsanity.com/paper/2607.00902/) (2026)

## Abstract (arXiv)

Existing temporal action detection (TAD) methods rely on generating an overwhelmingly large number of proposals per video. This leads to complex model designs due to proposal generation and/or per-proposal action instance evaluation and the resultant high computational cost. In this work, for the first time, we propose a proposal-free Temporal Action detection model with Global Segmentation mask (TAGS). Our core idea is to learn a global segmentation mask of each action instance jointly at the full video length. The TAGS model differs significantly from the conventional proposal-based methods by focusing on global temporal representation learning to directly detect local start and end points of action instances without proposals. Further, by modeling TAD holistically rather than locally at the individual proposal level, TAGS needs a much simpler model architecture with lower computational cost. Extensive experiments show that despite its simpler design, TAGS outperforms existing TAD methods, achieving new state-of-the-art performance on two benchmarks. Importantly, it is ~ 20x faster to train and ~1.6x more efficient for inference. Our PyTorch implementation of TAGS is available at https://github.com/sauradip/TAGS .