# A Structured Model For Action Detection

arXiv: https://arxiv.org/abs/1812.03544 · HTML: https://mlsanity.com/paper/1812.03544/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2018.

## Built on by (most cited first, 30 of 43)

- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [SiamMOT: Siamese Multi-Object Tracking](https://mlsanity.com/paper/2105.11595/) (2021)
- [Discovering Objects that Can Move](https://mlsanity.com/paper/2203.10159/) (2022)
- [Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs](https://mlsanity.com/paper/2305.06343/) (2023)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [Relational Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.04997/) (2023)
- [What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions](https://mlsanity.com/paper/2204.00746/) (2022)
- [Video-Mined Task Graphs for Keystep Recognition in Instructional Videos](https://mlsanity.com/paper/2307.08763/) (2023)
- [Probabilistic Representations for Video Contrastive Learning](https://mlsanity.com/paper/2204.03946/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [On the Benefits of 3D Pose and Tracking for Human Action Recognition](https://mlsanity.com/paper/2304.01199/) (2023)
- [Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2207.07783/) (2022)
- [Is an Object-Centric Video Representation Beneficial for Transfer?](https://mlsanity.com/paper/2207.10075/) (2022)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [End-to-End Spatio-Temporal Action Localisation with Video Transformers](https://mlsanity.com/paper/2304.12160/) (2023)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data](https://mlsanity.com/paper/2212.04821/) (2022)
- [Beyond Transfer Learning: Co-finetuning for Action Localisation](https://mlsanity.com/paper/2207.03807/) (2022)
- [Rethinking Image-to-Video Adaptation: An Object-centric Perspective](https://mlsanity.com/paper/2407.06871/) (2024)
- [A Simple and Efficient Pipeline to Build an End-to-End Spatial-Temporal Action Detector](https://mlsanity.com/paper/2206.03064/) (2022)
- [Game State and Spatio-temporal Action Detection in Soccer using Graph Neural Networks and 3D Convolutional Networks](https://mlsanity.com/paper/2502.15462/) (2025)
- [A Survey on Deep Learning-based Spatio-temporal Action Detection](https://mlsanity.com/paper/2308.01618/) (2023)
- [Learning Spatial-Temporal Graphs for Active Speaker Detection](https://mlsanity.com/paper/2112.01479/) (2021)
- [Multiscale Vision Transformers meet Bipartite Matching for efficient single-stage Action Localization](https://mlsanity.com/paper/2312.17686/) (2023)
- [VideoSAGE: Video Summarization with Graph Representation Learning](https://mlsanity.com/paper/2404.10539/) (2024)
- [Identity-aware Graph Memory Network for Action Detection](https://mlsanity.com/paper/2108.11559/) (2021)
- [CycleACR: Cycle Modeling of Actor-Context Relations for Video Action Detection](https://mlsanity.com/paper/2303.16118/) (2023)
- [OmViD: Omni-supervised active learning for video action detection](https://mlsanity.com/paper/2508.13983/) (2025)
- [LSTC: Boosting Atomic Action Detection with Long-Short-Term Context](https://mlsanity.com/paper/2110.09819/) (2021)

## Abstract (arXiv)

A dominant paradigm for learning-based approaches in computer vision is training generic models, such as ResNet for image recognition, or I3D for video understanding, on large datasets and allowing them to discover the optimal representation for the problem at hand. While this is an obviously attractive approach, it is not applicable in all scenarios. We claim that action detection is one such challenging problem - the models that need to be trained are large, and labeled data is expensive to obtain. To address this limitation, we propose to incorporate domain knowledge into the structure of the model, simplifying optimization. In particular, we augment a standard I3D network with a tracking module to aggregate long term motion patterns, and use a graph convolutional network to reason about interactions between actors and objects. Evaluated on the challenging AVA dataset, the proposed approach improves over the I3D baseline by 5.5% mAP and over the state-of-the-art by 4.8% mAP.