# What Actions are Needed for Understanding Human Actions in Videos?

arXiv: https://arxiv.org/abs/1708.02696 · HTML: https://mlsanity.com/paper/1708.02696/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2017.

## Built on by (most cited first, 29 of 29)

- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [Verbs in Action: Improving verb understanding in video-language models](https://mlsanity.com/paper/2304.06708/) (2023)
- [Test of Time: Instilling Video-Language Models with a Sense of Time](https://mlsanity.com/paper/2301.02074/) (2023)
- [Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding](https://mlsanity.com/paper/2103.16848/) (2021)
- [A Survey on Video Moment Localization](https://mlsanity.com/paper/2306.07515/) (2023)
- [HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction](https://mlsanity.com/paper/2412.13187/) (2024)
- [ELUDE: Generating interpretable explanations via a decomposition into labelled and unlabelled features](https://mlsanity.com/paper/2206.07690/) (2022)
- [Demystifying Unsupervised Semantic Correspondence Estimation](https://mlsanity.com/paper/2207.05054/) (2022)
- [Recur, Attend or Convolve? On Whether Temporal Modeling Matters for Cross-Domain Robustness in Action Recognition](https://mlsanity.com/paper/2112.12175/) (2021)
- [WhyAct: Identifying Action Reasons in Lifestyle Vlogs](https://mlsanity.com/paper/2109.02747/) (2021)
- [A Unified Taxonomy and Multimodal Dataset for Events in Invasion Games](https://mlsanity.com/paper/2108.11149/) (2021)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [Learning Joint Representation of Human Motion and Language](https://mlsanity.com/paper/2210.15187/) (2022)
- [Video BagNet: short temporal receptive fields increase robustness in long-term action recognition](https://mlsanity.com/paper/2308.11249/) (2023)
- [Black-box Error Diagnosis in Deep Neural Networks for Computer Vision: a Survey of Tools](https://mlsanity.com/paper/2201.06444/) (2022)
- [When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs](https://mlsanity.com/paper/2202.08138/) (2022)
- [Temporal Relevance Analysis for Video Action Models](https://mlsanity.com/paper/2204.11929/) (2022)
- [A Multi-stage deep architecture for summary generation of soccer videos](https://mlsanity.com/paper/2205.00694/) (2022)
- [No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration](https://mlsanity.com/paper/2211.00549/) (2022)
- [GELDA: A generative language annotation framework to reveal visual biases in datasets](https://mlsanity.com/paper/2311.18064/) (2023)
- [Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects](https://mlsanity.com/paper/2505.05318/) (2025)
- [Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognition](https://mlsanity.com/paper/2610.00064/) (2026)
- [Anomaly Recognition from surveillance videos using 3D Convolutional Neural Networks](https://mlsanity.com/paper/2101.01073/) (2021)
- [Real-Time Activity Recognition and Intention Recognition Using a Vision-based Embedded System](https://mlsanity.com/paper/2107.12744/) (2021)
- [AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval](https://mlsanity.com/paper/2203.16062/) (2022)
- [Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding](https://mlsanity.com/paper/2208.10077/) (2022)
- [Human Action Co-occurrence in Lifestyle Vlogs using Graph Link Prediction](https://mlsanity.com/paper/2309.06219/) (2023)
- [VidCoM: Fast Video Comprehension through Large Language Models with Multimodal Tools](https://mlsanity.com/paper/2310.10586/) (2023)
- [The impact of Compositionality in Zero-shot Multi-label action recognition for Object-based tasks](https://mlsanity.com/paper/2405.08695/) (2024)

## Abstract (arXiv)

What is the right way to reason about human activities? What directions forward are most promising? In this work, we analyze the current state of human activity understanding in videos. The goal of this paper is to examine datasets, evaluation metrics, algorithms, and potential future directions. We look at the qualitative attributes that define activities such as pose variability, brevity, and density. The experiments consider multiple state-of-the-art algorithms and multiple datasets. The results demonstrate that while there is inherent ambiguity in the temporal extent of activities, current datasets still permit effective benchmarking. We discover that fine-grained understanding of objects and pose when combined with temporal reasoning is likely to yield substantial improvements in algorithmic accuracy. We present the many kinds of information that will be needed to achieve substantial gains in activity understanding: objects, verbs, intent, and sequential reasoning. The software and additional information will be made available to provide other researchers detailed diagnostics to understand their own algorithms.