# Pointly-Supervised Action Localization

arXiv: https://arxiv.org/abs/1805.11333 · HTML: https://mlsanity.com/paper/1805.11333/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2018.

## Built on by (most cited first, 12 of 12)

- [Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2104.06404/) (2021)
- [Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2108.05029/) (2021)
- [Multi-modal Prompting for Low-Shot Temporal Action Localization](https://mlsanity.com/paper/2303.11732/) (2023)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [Semi-supervised Active Learning for Video Action Detection](https://mlsanity.com/paper/2312.07169/) (2023)
- [Smart Director: An Event-Driven Directing System for Live Broadcasting](https://mlsanity.com/paper/2201.04024/) (2022)
- [Point-VOS: Pointing Up Video Object Segmentation](https://mlsanity.com/paper/2402.05917/) (2024)
- [OmViD: Omni-supervised active learning for video action detection](https://mlsanity.com/paper/2508.13983/) (2025)
- [Exploiting Instance-based Mixed Sampling via Auxiliary Source Domain Supervision for Domain-adaptive Action Detection](https://mlsanity.com/paper/2209.15439/) (2022)
- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [Spot What Matters: Learning Context Using Graph Convolutional Networks for Weakly-Supervised Action Detection](https://mlsanity.com/paper/2107.13648/) (2021)

## Abstract (arXiv)

This paper strives for spatio-temporal localization of human actions in videos. In the literature, the consensus is to achieve localization by training on bounding box annotations provided for each frame of each training video. As annotating boxes in video is expensive, cumbersome and error-prone, we propose to bypass box-supervision. Instead, we introduce action localization based on point-supervision. We start from unsupervised spatio-temporal proposals, which provide a set of candidate regions in videos. While normally used exclusively for inference, we show spatio-temporal proposals can also be leveraged during training when guided by a sparse set of point annotations. We introduce an overlap measure between points and spatio-temporal proposals and incorporate them all into a new objective of a Multiple Instance Learning optimization. During inference, we introduce pseudo-points, visual cues from videos, that automatically guide the selection of spatio-temporal proposals. We outline five spatial and one temporal pseudo-point, as well as a measure to best leverage pseudo-points at test time. Experimental evaluation on three action localization datasets shows our pointly-supervised approach (i) is as effective as traditional box-supervision at a fraction of the annotation cost, (ii) is robust to sparse and noisy point annotations, (iii) benefits from pseudo-points during inference, and (iv) outperforms recent weakly-supervised alternatives. This leads us to conclude that points provide a viable alternative to boxes for action localization.