# Learning to Localize Actions from Moments

arXiv: https://arxiv.org/abs/2008.13705 · HTML: https://mlsanity.com/paper/2008.13705/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [Stand-Alone Inter-Frame Attention in Video Models](https://mlsanity.com/paper/2206.06931/) (2022)
- [BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation](https://mlsanity.com/paper/2305.18326/) (2023)
- [Representing Videos as Discriminative Sub-graphs for Action Recognition](https://mlsanity.com/paper/2201.04027/) (2022)
- [Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation](https://mlsanity.com/paper/2505.16985/) (2025)
- [Dynamic Temporal Filtering in Video Models](https://mlsanity.com/paper/2211.08252/) (2022)
- [Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization](https://mlsanity.com/paper/2407.07024/) (2024)
- [Spatiotemporal Deformable Scene Graphs for Complex Activity Detection](https://mlsanity.com/paper/2104.08194/) (2021)
- [Optimization Planning for 3D ConvNets](https://mlsanity.com/paper/2201.04021/) (2022)
- [Boosting Video Representation Learning with Multi-Faceted Integration](https://mlsanity.com/paper/2201.04023/) (2022)
- [ReLER@ZJU Submission to the Ego4D Moment Queries Challenge 2022](https://mlsanity.com/paper/2211.09558/) (2022)
- [Spatiotemporal Event Graphs for Dynamic Scene Understanding](https://mlsanity.com/paper/2312.07621/) (2023)
- [GTAutoAct: An Automatic Datasets Generation Framework Based on Game Engine Redevelopment for Action Recognition](https://mlsanity.com/paper/2401.13414/) (2024)

## Abstract (arXiv)

With the knowledge of action moments (i.e., trimmed video clips that each contains an action instance), humans could routinely localize an action temporally in an untrimmed video. Nevertheless, most practical methods still require all training videos to be labeled with temporal annotations (action category and temporal boundary) and develop the models in a fully-supervised manner, despite expensive labeling efforts and inapplicable to new categories. In this paper, we introduce a new design of transfer learning type to learn action localization for a large set of action categories, but only on action moments from the categories of interest and temporal annotations of untrimmed videos from a small set of action classes. Specifically, we present Action Herald Networks (AherNet) that integrate such design into an one-stage action localization framework. Technically, a weight transfer function is uniquely devised to build the transformation between classification of action moments or foreground video segments and action localization in synthetic contextual moments or untrimmed videos. The context of each moment is learnt through the adversarial mechanism to differentiate the generated features from those of background in untrimmed videos. Extensive experiments are conducted on the learning both across the splits of ActivityNet v1.3 and from THUMOS14 to ActivityNet v1.3. Our AherNet demonstrates the superiority even comparing to most fully-supervised action localization methods. More remarkably, we train AherNet to localize actions from 600 categories on the leverage of action moments in Kinetics-600 and temporal annotations from 200 classes in ActivityNet v1.3. Source code and data are available at urlhttps://github.com/FuchenUSTC/AherNet.