# Weakly-Supervised Action Localization by Generative Attention Modeling

arXiv: https://arxiv.org/abs/2003.12424 · HTML: https://mlsanity.com/paper/2003.12424/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2020.

## Built on by (most cited first, 30 of 59)

- [SimDA: Simple Diffusion Adapter for Efficient Video Generation](https://mlsanity.com/paper/2308.09710/) (2023)
- [Object-Region Video Transformers](https://mlsanity.com/paper/2110.06915/) (2021)
- [CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning](https://mlsanity.com/paper/2103.16392/) (2021)
- [SVFormer: Semi-supervised Video Transformer for Action Recognition](https://mlsanity.com/paper/2211.13222/) (2022)
- [MotionEditor: Editing Video Motion via Content-Aware Diffusion](https://mlsanity.com/paper/2311.18830/) (2023)
- [Implicit Temporal Modeling with Learnable Alignment for Video Recognition](https://mlsanity.com/paper/2304.10465/) (2023)
- [A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2101.00545/) (2021)
- [Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2108.05029/) (2021)
- [ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2203.15187/) (2022)
- [Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2107.12589/) (2021)
- [Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2203.16800/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Action Unit Memory Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2104.14135/) (2021)
- [Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2108.06524/) (2021)
- [Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation](https://mlsanity.com/paper/2103.16024/) (2021)
- [Weakly-Supervised Temporal Action Localization by Inferring Salient Snippet-Feature](https://mlsanity.com/paper/2303.12332/) (2023)
- [Weakly Supervised Action Selection Learning in Video](https://mlsanity.com/paper/2105.02439/) (2021)
- [Background-Click Supervision for Temporal Action Localization](https://mlsanity.com/paper/2111.12449/) (2021)
- [ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2103.15088/) (2021)
- [Proposal-Based Multiple Instance Learning for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2305.17861/) (2023)
- [ACM-Net: Action Context Modeling Network for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2104.02967/) (2021)
- [Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2111.05222/) (2021)
- [Multi-modal Prompting for Low-Shot Temporal Action Localization](https://mlsanity.com/paper/2303.11732/) (2023)
- [Adaptive Mutual Supervision for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2104.02357/) (2021)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [ACGNet: Action Complement Graph Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2112.10977/) (2021)
- [The Blessings of Unlabeled Background in Untrimmed Videos](https://mlsanity.com/paper/2103.13183/) (2021)
- [Distilling Vision-Language Pre-training to Collaborate with Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2212.09335/) (2022)
- [Deep Motion Prior for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2108.05607/) (2021)
- [DDG-Net: Discriminability-Driven Graph Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2307.16415/) (2023)

## Abstract (arXiv)

Weakly-supervised temporal action localization is a problem of learning an action localization model with only video-level action labeling available. The general framework largely relies on the classification activation, which employs an attention model to identify the action-related frames and then categorizes them into different classes. Such method results in the action-context confusion issue: context frames near action clips tend to be recognized as action frames themselves, since they are closely related to the specific classes. To solve the problem, in this paper we propose to model the class-agnostic frame-wise probability conditioned on the frame attention using conditional Variational Auto-Encoder (VAE). With the observation that the context exhibits notable difference from the action at representation level, a probabilistic model, i.e., conditional VAE, is learned to model the likelihood of each frame given the attention. By maximizing the conditional probability with respect to the attention, the action and non-action frames are well separated. Experiments on THUMOS14 and ActivityNet1.2 demonstrate advantage of our method and effectiveness in handling action-context confusion problem. Code is now available on GitHub.