# About Time: Advances, Challenges, and Outlooks of Action Understanding

arXiv: https://arxiv.org/abs/2411.15106 · HTML: https://mlsanity.com/paper/2411.15106/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2024.

## Built on by (most cited first, 6 of 6)

- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)
- [Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Survey](https://mlsanity.com/paper/2510.09731/) (2025)
- [HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence](https://mlsanity.com/paper/2608.26517/) (2026)
- [Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges](https://mlsanity.com/paper/2510.19292/) (2025)
- [TRANSPORTER: Transferring Visual Semantics from VLM Manifolds](https://mlsanity.com/paper/2511.18359/) (2025)
- [Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges](https://mlsanity.com/paper/2609.31665/) (2026)

## Abstract (arXiv)

We have witnessed impressive advances in video action understanding. Increased dataset sizes, variability, and computation availability have enabled leaps in performance and task diversification. Current systems can provide coarse- and fine-grained descriptions of video scenes, extract segments corresponding to queries, synthesize unobserved parts of videos, and predict context across multiple modalities. This survey comprehensively reviews advances in uni- and multi-modal action understanding across a range of tasks. We focus on prevalent challenges, overview widely adopted datasets, and survey seminal works with an emphasis on recent advances. We broadly distinguish between three temporal scopes: (1) recognition tasks of actions observed in full, (2) prediction tasks for ongoing partially observed actions, and (3) forecasting tasks for subsequent unobserved action(s). This division allows us to identify specific action modeling and video representation challenges. Finally, we outline future directions to address current shortcomings.