# ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization

arXiv: https://arxiv.org/abs/2103.15088 · HTML: https://mlsanity.com/paper/2103.15088/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 28 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals](https://mlsanity.com/paper/1703.06189/) (2017)
- [Weakly Supervised Action Localization by Sparse Temporal Pooling Network](https://mlsanity.com/paper/1712.05080/) (2017)
- [W-TALC: Weakly-supervised Temporal Activity Localization and Classification](https://mlsanity.com/paper/1807.10418/) (2018)
- [Multi-granularity Generator for Temporal Action Proposal](https://mlsanity.com/paper/1811.11524/) (2018)
- [AutoLoc: Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1807.08333/) (2018)
- [Background Suppression Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1911.09963/) (2019)
- [Weakly-Supervised Action Localization by Generative Attention Modeling](https://mlsanity.com/paper/2003.12424/) (2020)
- [Weakly-supervised Action Localization with Background Modeling](https://mlsanity.com/paper/1908.06552/) (2019)
- [Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks](https://mlsanity.com/paper/1510.00562/) (2015)
- [Marginalized Average Attentional Network for Weakly-Supervised Learning](https://mlsanity.com/paper/1905.08586/) (2019)
- [Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images](https://mlsanity.com/paper/1504.00983/) (2015)
- [Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/1908.02422/) (2019)
- [Review of Action Recognition and Detection Methods](https://mlsanity.com/paper/1610.06906/) (2016)

## Built on by (most cited first, 20 of 20)

- [Enriching Local and Global Contexts for Temporal Action Localization](https://mlsanity.com/paper/2107.12960/) (2021)
- [Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2107.12589/) (2021)
- [Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2203.16800/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Weakly Supervised Temporal Action Localization via Representative Snippet Knowledge Propagation](https://mlsanity.com/paper/2203.02925/) (2022)
- [Background-Click Supervision for Temporal Action Localization](https://mlsanity.com/paper/2111.12449/) (2021)
- [ACM-Net: Action Context Modeling Network for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2104.02967/) (2021)
- [Improving Weakly Supervised Temporal Action Localization by Bridging Train-Test Gap in Pseudo Labels](https://mlsanity.com/paper/2304.07978/) (2023)
- [DDG-Net: Discriminability-Driven Graph Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2307.16415/) (2023)
- [Bridge the Gap: From Weak to Full Supervision for Temporal Action Localization with PseudoFormer](https://mlsanity.com/paper/2504.14860/) (2025)
- [Weakly-guided Self-supervised Pretraining for Temporal Activity Detection](https://mlsanity.com/paper/2111.13675/) (2021)
- [Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning](https://mlsanity.com/paper/2206.11011/) (2022)
- [Forcing the Whole Video as Background: An Adversarial Learning Strategy for Weakly Temporal Action Localization](https://mlsanity.com/paper/2207.06659/) (2022)
- [Slow Motion Matters: A Slow Motion Enhanced Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2211.11324/) (2022)
- [Weakly-supervised Micro- and Macro-expression Spotting Based on Multi-level Consistency](https://mlsanity.com/paper/2305.02734/) (2023)
- [Visual Self-paced Iterative Learning for Unsupervised Temporal Action Localization](https://mlsanity.com/paper/2312.07384/) (2023)
- [Convex Combination Consistency between Neighbors for Weakly-supervised Action Localization](https://mlsanity.com/paper/2205.00400/) (2022)
- [JCDNet: Joint of Common and Definite phases Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2303.17294/) (2023)
- [Video-Specific Query-Key Attention Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2305.04186/) (2023)
- [EMA-SAM: Exponential Moving-average for SAM-based PTMC Segmentation](https://mlsanity.com/paper/2510.18213/) (2025)

## Abstract (arXiv)

The object of Weakly-supervised Temporal Action Localization (WS-TAL) is to localize all action instances in an untrimmed video with only video-level supervision. Due to the lack of frame-level annotations during training, current WS-TAL methods rely on attention mechanisms to localize the foreground snippets or frames that contribute to the video-level classification task. This strategy frequently confuse context with the actual action, in the localization result. Separating action and context is a core problem for precise WS-TAL, but it is very challenging and has been largely ignored in the literature. In this paper, we introduce an Action-Context Separation Network (ACSNet) that explicitly takes into account context for accurate action localization. It consists of two branches (i.e., the Foreground-Background branch and the Action-Context branch). The Foreground- Background branch first distinguishes foreground from background within the entire video while the Action-Context branch further separates the foreground as action and context. We associate video snippets with two latent components (i.e., a positive component and a negative component), and their different combinations can effectively characterize foreground, action and context. Furthermore, we introduce extended labels with auxiliary context categories to facilitate the learning of action-context separation. Experiments on THUMOS14 and ActivityNet v1.2/v1.3 datasets demonstrate the ACSNet outperforms existing state-of-the-art WS-TAL methods by a large margin.