# A flexible model for training action localization with varying levels of supervision

arXiv: https://arxiv.org/abs/1806.11328 · HTML: https://mlsanity.com/paper/1806.11328/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2018.

## Built on by (most cited first, 16 of 16)

- [Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models](https://mlsanity.com/paper/2305.19595/) (2023)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Video-Text Representation Learning via Differentiable Weak Temporal Alignment](https://mlsanity.com/paper/2203.16784/) (2022)
- [Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling](https://mlsanity.com/paper/2303.15270/) (2023)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [Constraint and Union for Partially-Supervised Temporal Sentence Grounding](https://mlsanity.com/paper/2302.09850/) (2023)
- [FETA: Towards Specializing Foundation Models for Expert Task Applications](https://mlsanity.com/paper/2209.03648/) (2022)
- [Semi-supervised Active Learning for Video Action Detection](https://mlsanity.com/paper/2312.07169/) (2023)
- [OmViD: Omni-supervised active learning for video action detection](https://mlsanity.com/paper/2508.13983/) (2025)
- [Boundary Discretization and Reliable Classification Network for Temporal Action Detection](https://mlsanity.com/paper/2310.06403/) (2023)
- ["Previously on ..." From Recaps to Story Summarization](https://mlsanity.com/paper/2405.11487/) (2024)
- [Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting](https://mlsanity.com/paper/2101.08567/) (2021)
- [Spot What Matters: Learning Context Using Graph Convolutional Networks for Weakly-Supervised Action Detection](https://mlsanity.com/paper/2107.13648/) (2021)
- [Learning from Synthetic Data via Provenance-Based Input Gradient Guidance](https://mlsanity.com/paper/2604.02946/) (2026)
- [Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining](https://mlsanity.com/paper/2608.25701/) (2026)

## Abstract (arXiv)

Spatio-temporal action detection in videos is typically addressed in a fully-supervised setup with manual annotation of training videos required at every frame. Since such annotation is extremely tedious and prohibits scalability, there is a clear need to minimize the amount of manual supervision. In this work we propose a unifying framework that can handle and combine varying types of less-demanding weak supervision. Our model is based on discriminative clustering and integrates different types of supervision as constraints on the optimization. We investigate applications of such a model to training setups with alternative supervisory signals ranging from video-level class labels to the full per-frame annotation of action bounding boxes. Experiments on the challenging UCF101-24 and DALY datasets demonstrate competitive performance of our method at a fraction of supervision used by previous methods. The flexibility of our model enables joint learning from data with different levels of annotation. Experimental results demonstrate a significant gain by adding a few fully supervised examples to otherwise weakly labeled videos.