# Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modeling

arXiv: https://arxiv.org/abs/2308.09946 · HTML: https://mlsanity.com/paper/2308.09946/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Weakly Supervised Action Localization by Sparse Temporal Pooling Network](https://mlsanity.com/paper/1712.05080/) (2017)
- [W-TALC: Weakly-supervised Temporal Activity Localization and Classification](https://mlsanity.com/paper/1807.10418/) (2018)
- [Background Suppression Network for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1911.09963/) (2019)
- [CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning](https://mlsanity.com/paper/2103.16392/) (2021)
- [Weakly-Supervised Action Localization by Generative Attention Modeling](https://mlsanity.com/paper/2003.12424/) (2020)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Weakly-supervised Action Localization with Background Modeling](https://mlsanity.com/paper/1908.06552/) (2019)
- [Two-Stream Consensus Network for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2010.11594/) (2020)
- [3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization](https://mlsanity.com/paper/1908.08216/) (2019)
- [A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2101.00545/) (2021)
- [MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection](https://mlsanity.com/paper/2007.09833/) (2020)
- [Clockwork Variational Autoencoders](https://mlsanity.com/paper/2102.09532/) (2021)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [Adversarial Background-Aware Loss for Weakly-supervised Temporal Activity Localization](https://mlsanity.com/paper/2007.06643/) (2020)
- [ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2203.15187/) (2022)
- [Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2203.16800/) (2022)
- [Weakly Supervised Temporal Action Localization via Representative Snippet Knowledge Propagation](https://mlsanity.com/paper/2203.02925/) (2022)
- [Graph Convolutional Module for Temporal Action Localization in Videos](https://mlsanity.com/paper/2112.00302/) (2021)
- [Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2108.06524/) (2021)
- [Marginalized Average Attentional Network for Weakly-Supervised Learning](https://mlsanity.com/paper/1905.08586/) (2019)

## Built on by (most cited first, 4 of 4)

- [Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models](https://mlsanity.com/paper/2411.08466/) (2024)
- [Action-Agnostic Point-Level Supervision for Temporal Action Detection](https://mlsanity.com/paper/2412.21205/) (2024)
- [Probabilistic Vision-Language Representation for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2408.05955/) (2024)
- [Generative Model-Based Feature Attention Module for Video Action Analysis](https://mlsanity.com/paper/2508.13565/) (2025)

## Abstract (arXiv)

Weakly-supervised action localization aims to recognize and localize action instancese in untrimmed videos with only video-level labels. Most existing models rely on multiple instance learning(MIL), where the predictions of unlabeled instances are supervised by classifying labeled bags. The MIL-based methods are relatively well studied with cogent performance achieved on classification but not on localization. Generally, they locate temporal regions by the video-level classification but overlook the temporal variations of feature semantics. To address this problem, we propose a novel attention-based hierarchically-structured latent model to learn the temporal variations of feature semantics. Specifically, our model entails two components, the first is an unsupervised change-points detection module that detects change-points by learning the latent representations of video features in a temporal hierarchy based on their rates of change, and the second is an attention-based classification model that selects the change-points of the foreground as the boundaries. To evaluate the effectiveness of our model, we conduct extensive experiments on two benchmark datasets, THUMOS-14 and ActivityNet-v1.3. The experiments show that our method outperforms current state-of-the-art methods, and even achieves comparable performance with fully-supervised methods.