# CTAP: Complementary Temporal Action Proposal Generation

arXiv: https://arxiv.org/abs/1807.04821 · HTML: https://mlsanity.com/paper/1807.04821/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2018.

## Built on by (most cited first, 30 of 46)

- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [Enriching Local and Global Contexts for Temporal Action Localization](https://mlsanity.com/paper/2107.12960/) (2021)
- [ReAct: Temporal Action Detection with Relational Queries](https://mlsanity.com/paper/2207.07097/) (2022)
- [Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization](https://mlsanity.com/paper/2305.01979/) (2023)
- [An Empirical Study of End-to-End Temporal Action Detection](https://mlsanity.com/paper/2204.02932/) (2022)
- [Self-Supervised Learning for Semi-Supervised Temporal Action Proposal](https://mlsanity.com/paper/2104.03214/) (2021)
- [DCAN: Improving Temporal Action Detection via Dual Context Aggregation](https://mlsanity.com/paper/2112.03612/) (2021)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Graph Convolutional Module for Temporal Action Localization in Videos](https://mlsanity.com/paper/2112.00302/) (2021)
- [Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation](https://mlsanity.com/paper/2103.16024/) (2021)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization](https://mlsanity.com/paper/2106.14118/) (2021)
- [Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization](https://mlsanity.com/paper/2103.15233/) (2021)
- [Multi-modal Prompting for Low-Shot Temporal Action Localization](https://mlsanity.com/paper/2303.11732/) (2023)
- [Adaptive Mutual Supervision for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2104.02357/) (2021)
- [Temporal Action Proposal Generation with Transformers](https://mlsanity.com/paper/2105.12043/) (2021)
- [RGB Stream Is Enough for Temporal Action Detection](https://mlsanity.com/paper/2107.04362/) (2021)
- [Temporal Action Proposal Generation with Background Constraint](https://mlsanity.com/paper/2112.07984/) (2021)
- [Few-Shot Temporal Action Localization with Query Adaptive Transformer](https://mlsanity.com/paper/2110.10552/) (2021)
- [AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation](https://mlsanity.com/paper/2210.02578/) (2022)
- [Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2212.09335/) (2022)
- [AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation](https://mlsanity.com/paper/2110.11474/) (2021)
- [ETAD: Training Action Detection End to End on a Laptop](https://mlsanity.com/paper/2205.07134/) (2022)
- [Agent-Environment Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/2107.08323/) (2021)
- [Smart Director: An Event-Driven Directing System for Live Broadcasting](https://mlsanity.com/paper/2201.04024/) (2022)
- [ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation](https://mlsanity.com/paper/2203.08942/) (2022)
- [Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization](https://mlsanity.com/paper/2408.01532/) (2024)

## Abstract (arXiv)

Temporal action proposal generation is an important task, akin to object proposals, temporal action proposals are intended to capture "clips" or temporal intervals in videos that are likely to contain an action. Previous methods can be divided to two groups: sliding window ranking and actionness score grouping. Sliding windows uniformly cover all segments in videos, but the temporal boundaries are imprecise; grouping based method may have more precise boundaries but it may omit some proposals when the quality of actionness score is low. Based on the complementary characteristics of these two methods, we propose a novel Complementary Temporal Action Proposal (CTAP) generator. Specifically, we apply a Proposal-level Actionness Trustworthiness Estimator (PATE) on the sliding windows proposals to generate the probabilities indicating whether the actions can be correctly detected by actionness scores, the windows with high scores are collected. The collected sliding windows and actionness proposals are then processed by a temporal convolutional neural network for proposal ranking and boundary adjustment. CTAP outperforms state-of-the-art methods on average recall (AR) by a large margin on THUMOS-14 and ActivityNet 1.3 datasets. We further apply CTAP as a proposal generation method in an existing action detector, and show consistent significant improvements.