# ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition

arXiv: https://arxiv.org/abs/2410.05774 · HTML: https://mlsanity.com/paper/2410.05774/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2024.

## Built on by (most cited first, 21 of 21)

- [Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs](https://mlsanity.com/paper/2505.11842/) (2025)
- [SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding](https://mlsanity.com/paper/2505.16630/) (2025)
- [DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2511.12908/) (2025)
- [ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment](https://mlsanity.com/paper/2506.22967/) (2025)
- [The Escalator Problem: Identifying Implicit Motion Blindness in AI for Accessibility](https://mlsanity.com/paper/2508.07989/) (2025)
- [HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes](https://mlsanity.com/paper/2508.13692/) (2025)
- [The Potential and Limitations of Vision-Language Models for Human Motion Understanding: A Case Study in Data-Driven Stroke Rehabilitation](https://mlsanity.com/paper/2511.17727/) (2025)
- [HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics](https://mlsanity.com/paper/2512.00885/) (2025)
- [Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs](https://mlsanity.com/paper/2605.22823/) (2026)
- [EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footage](https://mlsanity.com/paper/2607.06468/) (2026)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [ViTs for Action Classification in Videos: An Approach to Risky Tackle Detection in American Football Practice Videos](https://mlsanity.com/paper/2604.01318/) (2026)
- [TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions](https://mlsanity.com/paper/2604.27975/) (2026)
- [FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models](https://mlsanity.com/paper/2608.08736/) (2026)
- [DMC3: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering](https://mlsanity.com/paper/2510.20285/) (2025)
- [VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition](https://mlsanity.com/paper/2605.02834/) (2026)
- [BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding](https://mlsanity.com/paper/2605.12074/) (2026)
- [Jailbreaking Multimodal Large Language Models using Multi-Clip Video](https://mlsanity.com/paper/2606.02111/) (2026)
- [SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoning](https://mlsanity.com/paper/2608.07932/) (2026)
- [A Survey of Large Models in Sports](https://mlsanity.com/paper/2608.14377/) (2026)
- [Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining](https://mlsanity.com/paper/2608.25701/) (2026)

## Abstract (arXiv)

Our world is full of varied actions and moves across specialized domains that we, as humans, strive to identify and understand. Within any single domain, actions can often appear quite similar, making it challenging for deep models to distinguish them accurately. To evaluate the effectiveness of multimodal foundation models in helping us recognize such actions, we present ActionAtlas v1.0, a multiple-choice video question answering benchmark featuring short videos across various sports. Each video in the dataset is paired with a question and four or five choices. The question pinpoints specific individuals, asking which choice "best" describes their action within a certain temporal context. Overall, the dataset includes 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices. Unlike most existing video question answering benchmarks that only cover simplistic actions, often identifiable from a single frame, ActionAtlas focuses on intricate movements and rigorously tests the model's capability to discern subtle differences between moves that look similar within each domain. We evaluate open and proprietary foundation models on this benchmark, finding that the best model, GPT-4o, achieves a maximum accuracy of 45.52%. Meanwhile, Non-expert crowd workers, provided with action description for each choice, achieve 61.64% accuracy, where random chance is approximately 21%. Our findings with state-of-the-art models indicate that having a high frame sampling rate is important for accurately recognizing actions in ActionAtlas, a feature that some leading proprietary video models, such as Gemini, do not include in their default configuration.