# Video Action Detection: Analysing Limitations and Challenges

arXiv: https://arxiv.org/abs/2204.07892 · HTML: https://mlsanity.com/paper/2204.07892/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 17 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [EnD: Entangling and Disentangling deep representations for bias correction](https://mlsanity.com/paper/2103.02023/) (2021)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)
- [STEP: Spatio-Temporal Progressive Learning for Video Action Detection](https://mlsanity.com/paper/1904.09288/) (2019)
- [Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos](https://mlsanity.com/paper/1703.10664/) (2017)
- [Learning to track for spatio-temporal action localization](https://mlsanity.com/paper/1506.01929/) (2015)
- [MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection](https://mlsanity.com/paper/2012.00914/) (2020)
- [VideoCapsuleNet: A Simplified Network for Action Detection](https://mlsanity.com/paper/1805.08162/) (2018)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [TinyVIRAT: Low-resolution Video Action Recognition](https://mlsanity.com/paper/2007.07355/) (2020)
- [Gabriella: An Online System for Real-Time Activity Detection in Untrimmed Security Videos](https://mlsanity.com/paper/2004.11475/) (2020)
- [Argus++: Robust Real-time Activity Detection for Unconstrained Video Streams with Overlapping Cube Proposals](https://mlsanity.com/paper/2201.05290/) (2022)
- [We don't Need Thousand Proposalscolon Single Shot Actor-Action Detection in Videos](https://mlsanity.com/paper/2011.10927/) (2020)
- [Real-Time End-to-End Action Detection with Two-Stream Networks](https://mlsanity.com/paper/1802.08362/) (2018)
- [TinyAction Challenge: Recognizing Real-world Low-resolution Activities in Videos](https://mlsanity.com/paper/2107.11494/) (2021)

## Built on by (most cited first, 12 of 12)

- [Foundation Models for Video Understanding: A Survey](https://mlsanity.com/paper/2405.03770/) (2024)
- [STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding](https://mlsanity.com/paper/2502.20678/) (2025)
- [Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2501.17053/) (2025)
- [Described Spatial-Temporal Video Detection](https://mlsanity.com/paper/2407.05610/) (2024)
- [A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2504.06153/) (2025)
- [OmViD: Omni-supervised active learning for video action detection](https://mlsanity.com/paper/2508.13983/) (2025)
- [Asynchronous Perception Machine For Efficient Test-Time-Training](https://mlsanity.com/paper/2410.20535/) (2024)
- [Punching Bag vs. Punching Person: Motion Transferability in Videos](https://mlsanity.com/paper/2508.00085/) (2025)
- [Scaling Open-Vocabulary Action Detection](https://mlsanity.com/paper/2504.03096/) (2025)
- [GaitCrafter: Diffusion Model for Biometric Preserving Gait Synthesis](https://mlsanity.com/paper/2508.13300/) (2025)
- [RobustGait: Robustness Analysis for Appearance Based Gait Recognition](https://mlsanity.com/paper/2511.13065/) (2025)
- [VISTA: Video Interaction Spatio-Temporal Analysis Benchmark](https://mlsanity.com/paper/2605.01391/) (2026)

## Abstract (arXiv)

Beyond possessing large enough size to feed data hungry machines (eg, transformers), what attributes measure the quality of a dataset? Assuming that the definitions of such attributes do exist, how do we quantify among their relative existences? Our work attempts to explore these questions for video action detection. The task aims to spatio-temporally localize an actor and assign a relevant action class. We first analyze the existing datasets on video action detection and discuss their limitations. Next, we propose a new dataset, Multi Actor Multi Action (MAMA) which overcomes these limitations and is more suitable for real world applications. In addition, we perform a biasness study which analyzes a key property differentiating videos from static images: the temporal aspect. This reveals if the actions in these datasets really need the motion information of an actor, or whether they predict the occurrence of an action even by looking at a single frame. Finally, we investigate the widely held assumptions on the importance of temporal ordering: is temporal ordering important for detecting these actions? Such extreme experiments show existence of biases which have managed to creep into existing methods inspite of careful modeling.