# Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization

arXiv: https://arxiv.org/abs/2006.07976 · HTML: https://mlsanity.com/paper/2006.07976/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2020.

## Built on by (most cited first, 30 of 91)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [ROAD: The ROad event Awareness Dataset for Autonomous Driving](https://mlsanity.com/paper/2102.11585/) (2021)
- [InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges](https://mlsanity.com/paper/2211.09529/) (2022)
- [Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts](https://mlsanity.com/paper/2408.05905/) (2024)
- [Looking Backward: Streaming Video-to-Video Translation with Feature Banks](https://mlsanity.com/paper/2405.15757/) (2024)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [On the Benefits of 3D Pose and Tracking for Human Action Recognition](https://mlsanity.com/paper/2304.01199/) (2023)
- [UniCon: Unified Context Network for Robust Active Speaker Detection](https://mlsanity.com/paper/2108.02607/) (2021)
- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [Open Set Action Recognition via Multi-Label Evidential Learning](https://mlsanity.com/paper/2303.12698/) (2023)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2303.15879/) (2023)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [End-to-End Spatio-Temporal Action Localisation with Video Transformers](https://mlsanity.com/paper/2304.12160/) (2023)
- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling](https://mlsanity.com/paper/2303.15270/) (2023)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos](https://mlsanity.com/paper/2404.04565/) (2024)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [MissionGNN: Hierarchical Multimodal GNN-based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation](https://mlsanity.com/paper/2406.18815/) (2024)
- [Panoramic Human Activity Recognition](https://mlsanity.com/paper/2203.03806/) (2022)

## Abstract (arXiv)

Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling direct pairwise relations between entities. In this paper, we take one step further, not only model direct relations between pairs but also take into account indirect higher-order relations established upon multiple elements. We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context. To this end, we design an Actor-Context-Actor Relation Network (ACAR-Net) which builds upon a novel High-order Relation Reasoning Operator and an Actor-Context Feature Bank to enable indirect relation reasoning for spatio-temporal action localization. Experiments on AVA and UCF101-24 datasets show the advantages of modeling actor-context-actor relations, and visualization of attention maps further verifies that our model is capable of finding relevant higher-order relations to support action detection. Notably, our method ranks first in the AVA-Kineticsaction localization task of ActivityNet Challenge 2020, out-performing other entries by a significant margin (+6.71mAP). Training code and models will be available at https://github.com/Siyu-C/ACAR-Net.