# Intention-Conditioned Long-Term Human Egocentric Action Forecasting

arXiv: https://arxiv.org/abs/2207.12080 · HTML: https://mlsanity.com/paper/2207.12080/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [Future Person Localization in First-Person Videos](https://mlsanity.com/paper/1711.11217/) (2017)
- [LOKI: Long Term and Key Intentions for Trajectory Prediction](https://mlsanity.com/paper/2108.08236/) (2021)
- [Learning the Predictability of the Future](https://mlsanity.com/paper/2101.01600/) (2021)
- [Uncertainty-Aware Anticipation of Activities](https://mlsanity.com/paper/1908.09540/) (2019)
- [A Variational Auto-Encoder Model for Stochastic Point Processes](https://mlsanity.com/paper/1904.03273/) (2019)
- [Forecasting future action sequences with attention: a new approach to weakly supervised action forecasting](https://mlsanity.com/paper/1912.04608/) (2019)
- [Video + CLIP Baseline for Ego4D Long-term Action Anticipation](https://mlsanity.com/paper/2207.00579/) (2022)

## Built on by (most cited first, 30 of 35)

- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation](https://mlsanity.com/paper/2408.16730/) (2024)
- [Interaction Region Visual Transformer for Egocentric Action Anticipation](https://mlsanity.com/paper/2211.14154/) (2022)
- [PDPP: Projected Diffusion for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2303.14676/) (2023)
- [Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models](https://mlsanity.com/paper/2405.20305/) (2024)
- [PALM: Predicting Actions through Language Models](https://mlsanity.com/paper/2311.17944/) (2023)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views](https://mlsanity.com/paper/2405.13659/) (2024)
- [Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023](https://mlsanity.com/paper/2306.16545/) (2023)
- [Predicting the Next Action by Modeling the Abstract Goal](https://mlsanity.com/paper/2209.05044/) (2022)
- [Action Anticipation with Goal Consistency](https://mlsanity.com/paper/2306.15045/) (2023)
- [HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs](https://mlsanity.com/paper/2309.16524/) (2023)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos](https://mlsanity.com/paper/2411.02570/) (2024)
- [DiffAnt: Diffusion Models for Action Anticipation](https://mlsanity.com/paper/2311.15991/) (2023)
- [A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives](https://mlsanity.com/paper/2403.03037/) (2024)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Action Anticipation from SoccerNet Football Video Broadcasts](https://mlsanity.com/paper/2504.12021/) (2025)
- [Bidirectional Progressive Transformer for Interaction Intention Anticipation](https://mlsanity.com/paper/2405.05552/) (2024)
- [Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation](https://mlsanity.com/paper/2407.11954/) (2024)
- [Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectives](https://mlsanity.com/paper/2502.02487/) (2025)
- [Enhancing Visual Planning with Auxiliary Tasks and Multi-token Prediction](https://mlsanity.com/paper/2507.15130/) (2025)
- [VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipation](https://mlsanity.com/paper/2307.03918/) (2023)
- [MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipation](https://mlsanity.com/paper/2501.08837/) (2025)
- [Zero-Shot Temporal Interaction Localization for Egocentric Videos](https://mlsanity.com/paper/2506.03662/) (2025)
- [Optimizing Multitask Industrial Processes with Predictive Action Guidance](https://mlsanity.com/paper/2501.05108/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)
- [MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipation](https://mlsanity.com/paper/2509.11394/) (2025)
- [FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration](https://mlsanity.com/paper/2606.14778/) (2026)

## Abstract (arXiv)

To anticipate how a human would act in the future, it is essential to understand the human intention since it guides the human towards a certain goal. In this paper, we propose a hierarchical architecture which assumes a sequence of human action (low-level) can be driven from the human intention (high-level). Based on this, we deal with Long-Term Action Anticipation task in egocentric videos. Our framework first extracts two level of human information over the N observed videos human actions through a Hierarchical Multi-task MLP Mixer (H3M). Then, we condition the uncertainty of the future through an Intention-Conditioned Variational Auto-Encoder (I-CVAE) that generates K stable predictions of the next Z=20 actions that the observed human might perform. By leveraging human intention as high-level information, we claim that our model is able to anticipate more time-consistent actions in the long-term, thus improving the results over baseline methods in EGO4D Challenge. This work ranked first in both CVPR@2022 and ECVV@2022 EGO4D LTA Challenge by providing more plausible anticipated sequences, improving the anticipation of nouns and overall actions. Webpage: https://evm7.github.io/icvae-page/