# Actor-Centric Relation Network

arXiv: https://arxiv.org/abs/1807.10982 · HTML: https://mlsanity.com/paper/1807.10982/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2018.

## Built on by (most cited first, 30 of 87)

- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Object-Region Video Transformers](https://mlsanity.com/paper/2110.06915/) (2021)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs](https://mlsanity.com/paper/2305.06343/) (2023)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices](https://mlsanity.com/paper/2201.12382/) (2022)
- [AMEGO: Active Memory from long EGOcentric videos](https://mlsanity.com/paper/2409.10917/) (2024)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Helping Hands: An Object-Aware Ego-Centric Video Recognition Model](https://mlsanity.com/paper/2308.07918/) (2023)
- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [Open Set Action Recognition via Multi-Label Evidential Learning](https://mlsanity.com/paper/2303.12698/) (2023)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Is an Object-Centric Video Representation Beneficial for Transfer?](https://mlsanity.com/paper/2207.10075/) (2022)
- [ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors](https://mlsanity.com/paper/2310.16447/) (2023)
- [Object-centric Video Representation for Long-term Action Anticipation](https://mlsanity.com/paper/2311.00180/) (2023)
- [Understanding Video Transformers via Universal Concept Discovery](https://mlsanity.com/paper/2401.10831/) (2024)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2303.15879/) (2023)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [End-to-End Spatio-Temporal Action Localisation with Video Transformers](https://mlsanity.com/paper/2304.12160/) (2023)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [Hierarchical Self-supervised Representation Learning for Movie Understanding](https://mlsanity.com/paper/2204.03101/) (2022)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data](https://mlsanity.com/paper/2212.04821/) (2022)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations](https://mlsanity.com/paper/2105.07085/) (2021)

## Abstract (arXiv)

Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level and model temporal context with 3D ConvNets. Here, we go one step further and model spatio-temporal relations to capture the interactions between human actors, relevant objects and scene elements essential to differentiate similar human actions. Our approach is weakly supervised and mines the relevant elements automatically with an actor-centric relational network (ACRN). ACRN computes and accumulates pair-wise relation information from actor and global scene features, and generates relation features for action classification. It is implemented as neural networks and can be trained jointly with an existing action detection system. We show that ACRN outperforms alternative approaches which capture relation information, and that the proposed framework improves upon the state-of-the-art performance on JHMDB and AVA. A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.