# Actor-centered Representations for Action Localization in Streaming Videos

arXiv: https://arxiv.org/abs/2104.14131 · HTML: https://mlsanity.com/paper/2104.14131/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics](https://mlsanity.com/paper/1904.03597/) (2019)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)
- [VideoLSTM Convolves, Attends and Flows for Action Recognition](https://mlsanity.com/paper/1607.01794/) (2016)
- [Finding Action Tubes](https://mlsanity.com/paper/1411.6031/) (2014)
- [Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos](https://mlsanity.com/paper/1703.10664/) (2017)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Learning to track for spatio-temporal action localization](https://mlsanity.com/paper/1506.01929/) (2015)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [CERN: Confidence-Energy Recurrent Network for Group Activity Recognition](https://mlsanity.com/paper/1704.03058/) (2017)
- [Guess Where? Actor-Supervision for Spatiotemporal Action Localization](https://mlsanity.com/paper/1804.01824/) (2018)
- [Action Localization through Continual Predictive Learning](https://mlsanity.com/paper/2003.12185/) (2020)
- [Tubelets: Unsupervised action proposals from spatiotemporal super-voxels](https://mlsanity.com/paper/1607.02003/) (2016)
- [Going Deeper with Semantics: Video Activity Interpretation using Semantic Contextualization](https://mlsanity.com/paper/1708.03725/) (2017)

## Built on by (most cited first, 3 of 3)

- [Discovering Novel Actions from Open World Egocentric Videos with Object-Grounded Visual Commonsense Reasoning](https://mlsanity.com/paper/2305.16602/) (2023)
- [Self-supervised Multi-actor Social Activity Understanding in Streaming Videos](https://mlsanity.com/paper/2406.14472/) (2024)
- [Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learning](https://mlsanity.com/paper/2512.04219/) (2025)

## Abstract (arXiv)

Event perception tasks such as recognizing and localizing actions in streaming videos are essential for scaling to real-world application contexts. We tackle the problem of learning actor-centered representations through the notion of continual hierarchical predictive learning to localize actions in streaming videos without the need for training labels and outlines for the objects in the video. We propose a framework driven by the notion of hierarchical predictive learning to construct actor-centered features by attention-based contextualization. The key idea is that predictable features or objects do not attract attention and hence do not contribute to the action of interest. Experiments on three benchmark datasets show that the approach can learn robust representations for localizing actions using only one epoch of training, i.e., a single pass through the streaming video. We show that the proposed approach outperforms unsupervised and weakly supervised baselines while offering competitive performance to fully supervised approaches. Additionally, we extend the model to multi-actor settings to recognize group activities while localizing the multiple, plausible actors. We also show that it generalizes to out-of-domain data with limited performance degradation.