# When will you do what? - Anticipating Temporal Occurrences of Activities

arXiv: https://arxiv.org/abs/1804.00892 · HTML: https://mlsanity.com/paper/1804.00892/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2018.

## Built on by (most cited first, 30 of 107)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [HierVL: Learning Hierarchical Video-Language Embeddings](https://mlsanity.com/paper/2301.02311/) (2023)
- [Future Transformer for Long-term Action Anticipation](https://mlsanity.com/paper/2205.14022/) (2022)
- [E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition](https://mlsanity.com/paper/2112.03596/) (2021)
- [Learning the Predictability of the Future](https://mlsanity.com/paper/2101.01600/) (2021)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory](https://mlsanity.com/paper/2301.00746/) (2023)
- [HRP: Human Affordances for Robotic Pre-Training](https://mlsanity.com/paper/2407.18911/) (2024)
- [Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation](https://mlsanity.com/paper/2210.12649/) (2022)
- [Learning to Anticipate Egocentric Actions by Imagination](https://mlsanity.com/paper/2101.04924/) (2021)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Anticipating human actions by correlating past with the future with Jaccard similarity measures](https://mlsanity.com/paper/2105.12414/) (2021)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [Intention-Conditioned Long-Term Human Egocentric Action Forecasting](https://mlsanity.com/paper/2207.12080/) (2022)
- [Video-Mined Task Graphs for Keystep Recognition in Instructional Videos](https://mlsanity.com/paper/2307.08763/) (2023)
- [Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models](https://mlsanity.com/paper/2405.20305/) (2024)
- [Self-Regulated Learning for Egocentric Video Activity Anticipation](https://mlsanity.com/paper/2111.11631/) (2021)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [Rethinking Learning Approaches for Long-Term Action Anticipation](https://mlsanity.com/paper/2210.11566/) (2022)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset](https://mlsanity.com/paper/2301.03213/) (2023)
- [Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction](https://mlsanity.com/paper/2205.15608/) (2022)

## Abstract (arXiv)

Analyzing human actions in videos has gained increased attention recently. While most works focus on classifying and labeling observed video frames or anticipating the very recent future, making long-term predictions over more than just a few seconds is a task with many practical applications that has not yet been addressed. In this paper, we propose two methods to predict a considerably large amount of future actions and their durations. Both, a CNN and an RNN are trained to learn future video labels based on previously seen content. We show that our methods generate accurate predictions of the future even for long videos with a huge amount of different actions and can even deal with noisy or erroneous input information.