# Large Scale Holistic Video Understanding

arXiv: https://arxiv.org/abs/1904.11451 · HTML: https://mlsanity.com/paper/1904.11451/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 12 of 12)

- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos](https://mlsanity.com/paper/2203.14456/) (2022)
- [Mitigating Representation Bias in Action Recognition: Algorithms and Benchmarks](https://mlsanity.com/paper/2209.09393/) (2022)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization](https://mlsanity.com/paper/2411.02851/) (2024)
- [An Animation-based Augmentation Approach for Action Recognition from Discontinuous Video](https://mlsanity.com/paper/2404.06741/) (2024)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [Beyond still images: Temporal features and input variance resilience](https://mlsanity.com/paper/2311.00800/) (2023)
- [Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding](https://mlsanity.com/paper/2208.10077/) (2022)
- [GTAutoAct: An Automatic Datasets Generation Framework Based on Game Engine Redevelopment for Action Recognition](https://mlsanity.com/paper/2401.13414/) (2024)
- [Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis](https://mlsanity.com/paper/2502.07277/) (2025)
- [USV: Towards Understanding the User-generated Short-form Videos](https://mlsanity.com/paper/2605.20838/) (2026)

## Abstract (arXiv)

Video recognition has been advanced in recent years by benchmarks with rich annotations. However, research is still mainly limited to human action or sports recognition - focusing on a highly specific video understanding task and thus leaving a significant gap towards describing the overall content of a video. We fill this gap by presenting a large-scale "Holistic Video Understanding Dataset"~(HVU). HVU is organized hierarchically in a semantic taxonomy that focuses on multi-label and multi-task video understanding as a comprehensive problem that encompasses the recognition of multiple semantic aspects in the dynamic scene. HVU contains approx.~572k videos in total with 9 million annotations for training, validation, and test set spanning over 3142 labels. HVU encompasses semantic aspects defined on categories of scenes, objects, actions, events, attributes, and concepts which naturally captures the real-world scenarios. We demonstrate the generalization capability of HVU on three challenging tasks: 1.) Video classification, 2.) Video captioning and 3.) Video clustering tasks. In particular for video classification, we introduce a new spatio-temporal deep neural network architecture called "Holistic Appearance and Temporal Network"~(HATNet) that builds on fusing 2D and 3D architectures into one by combining intermediate representations of appearance and temporal cues. HATNet focuses on the multi-label and multi-task learning problem and is trained in an end-to-end manner. Via our experiments, we validate the idea that holistic representation learning is complementary, and can play a key role in enabling many real-world applications.