# Action Anticipation with Goal Consistency

arXiv: https://arxiv.org/abs/2306.15045 · HTML: https://mlsanity.com/paper/2306.15045/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Future Transformer for Long-term Action Anticipation](https://mlsanity.com/paper/2205.14022/) (2022)
- [Real-time Online Video Detection with Temporal Smoothing Transformers](https://mlsanity.com/paper/2209.09236/) (2022)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture](https://mlsanity.com/paper/1509.05016/) (2015)
- [Intention-Conditioned Long-Term Human Egocentric Action Forecasting](https://mlsanity.com/paper/2207.12080/) (2022)
- [Rethinking Learning Approaches for Long-Term Action Anticipation](https://mlsanity.com/paper/2210.11566/) (2022)
- [Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videos](https://mlsanity.com/paper/2107.09504/) (2021)
- [Self-supervised Learning for Unintentional Action Prediction](https://mlsanity.com/paper/2209.12074/) (2022)

## Built on by (most cited first, 12 of 12)

- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding](https://mlsanity.com/paper/2504.13915/) (2025)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation](https://mlsanity.com/paper/2407.11954/) (2024)
- [Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context](https://mlsanity.com/paper/2410.21275/) (2024)
- [Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?](https://mlsanity.com/paper/2512.02846/) (2025)
- [MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipation](https://mlsanity.com/paper/2501.08837/) (2025)
- [Learning to Visually Connect Actions and their Effects](https://mlsanity.com/paper/2401.10805/) (2024)
- [Understanding Multimodal Complementarity for Single-Frame Action Anticipation](https://mlsanity.com/paper/2601.22039/) (2026)
- [MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipation](https://mlsanity.com/paper/2509.11394/) (2025)
- [Anticipation through Head Pose Estimation: a preliminary study](https://mlsanity.com/paper/2408.05516/) (2024)

## Abstract (arXiv)

In this paper, we address the problem of short-term action anticipation, i.e., we want to predict an upcoming action one second before it happens. We propose to harness high-level intent information to anticipate actions that will take place in the future. To this end, we incorporate an additional goal prediction branch into our model and propose a consistency loss function that encourages the anticipated actions to conform to the high-level goal pursued in the video. In our experiments, we show the effectiveness of the proposed approach and demonstrate that our method achieves state-of-the-art results on two large-scale datasets: Assembly101 and COIN.