# Self-supervised Multi-actor Social Activity Understanding in Streaming Videos

arXiv: https://arxiv.org/abs/2406.14472 · HTML: https://mlsanity.com/paper/2406.14472/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)
- [GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer](https://mlsanity.com/paper/2108.12630/) (2021)
- [Social Adaptive Module for Weakly-supervised Group Activity Recognition](https://mlsanity.com/paper/2007.09470/) (2020)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos](https://mlsanity.com/paper/2007.02632/) (2020)
- [Convolutional Relational Machine for Group Activity Recognition](https://mlsanity.com/paper/1904.03308/) (2019)
- [Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition](https://mlsanity.com/paper/2204.02148/) (2022)
- [CERN: Confidence-Energy Recurrent Network for Group Activity Recognition](https://mlsanity.com/paper/1704.03058/) (2017)
- [Hunting Group Clues with Transformers for Social Group Activity Recognition](https://mlsanity.com/paper/2207.05254/) (2022)
- [Self-Feedback DETR for Temporal Action Detection](https://mlsanity.com/paper/2308.10570/) (2023)
- [Detector-Free Weakly Supervised Group Activity Recognition](https://mlsanity.com/paper/2204.02139/) (2022)
- [COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality](https://mlsanity.com/paper/2112.05892/) (2021)
- [SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition](https://mlsanity.com/paper/2305.06310/) (2023)
- [Action Localization through Continual Predictive Learning](https://mlsanity.com/paper/2003.12185/) (2020)
- [Towards Active Vision for Action Localization with Reactive Control and Predictive Learning](https://mlsanity.com/paper/2111.05448/) (2021)
- [Actor-centered Representations for Action Localization in Streaming Videos](https://mlsanity.com/paper/2104.14131/) (2021)
- [Group Activity Recognition using Unreliable Tracked Pose](https://mlsanity.com/paper/2401.03262/) (2024)

## Built on by (most cited first, 2 of 2)

- [EASE: Embodied Active Event Perception via Self-Supervised Energy Minimization](https://mlsanity.com/paper/2506.17516/) (2025)
- [Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learning](https://mlsanity.com/paper/2512.04219/) (2025)

## Abstract (arXiv)

This work addresses the problem of Social Activity Recognition (SAR), a critical component in real-world tasks like surveillance and assistive robotics. Unlike traditional event understanding approaches, SAR necessitates modeling individual actors' appearance and motions and contextualizing them within their social interactions. Traditional action localization methods fall short due to their single-actor, single-action assumption. Previous SAR research has relied heavily on densely annotated data, but privacy concerns limit their applicability in real-world settings. In this work, we propose a self-supervised approach based on multi-actor predictive learning for SAR in streaming videos. Using a visual-semantic graph structure, we model social interactions, enabling relational reasoning for robust performance with minimal labeled data. The proposed framework achieves competitive performance on standard group activity recognition benchmarks. Evaluation on three publicly available action localization benchmarks demonstrates its generalizability to arbitrary action localization.