# Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos

arXiv: https://arxiv.org/abs/2309.05943 · HTML: https://mlsanity.com/paper/2309.05943/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention](https://mlsanity.com/paper/1905.09035/) (2019)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [The More You Know: Using Knowledge Graphs for Image Classification](https://mlsanity.com/paper/1612.04844/) (2016)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Future Transformer for Long-term Action Anticipation](https://mlsanity.com/paper/2205.14022/) (2022)
- [Zero-Shot Anticipation for Instructional Activities](https://mlsanity.com/paper/1812.02501/) (2018)
- [Predicting the Future: A Jointly Learnt Model for Action Anticipation](https://mlsanity.com/paper/1912.07148/) (2019)
- [Long-Term Anticipation of Activities with Cycle Consistency](https://mlsanity.com/paper/2009.01142/) (2020)
- [Uncertainty-Aware Anticipation of Activities](https://mlsanity.com/paper/1908.09540/) (2019)
- [Encouraging LSTMs to Anticipate Actions Very Early](https://mlsanity.com/paper/1703.07023/) (2017)
- [Anticipating human actions by correlating past with the future with Jaccard similarity measures](https://mlsanity.com/paper/2105.12414/) (2021)
- [What can I do here? A Theory of Affordances in Reinforcement Learning](https://mlsanity.com/paper/2006.15085/) (2020)
- [FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations](https://mlsanity.com/paper/2203.16639/) (2022)
- [Video Action Understanding](https://mlsanity.com/paper/2010.06647/) (2020)
- [Sample-Efficient Learning of Novel Visual Concepts](https://mlsanity.com/paper/2306.09482/) (2023)
- [Text-Derived Knowledge Helps Vision: A Simple Cross-modal Distillation for Video-based Action Anticipation](https://mlsanity.com/paper/2210.05991/) (2022)
- [Visual Concept Reasoning Networks](https://mlsanity.com/paper/2008.11783/) (2020)

## Built on by (most cited first, 4 of 4)

- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)
- [ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decomposition](https://mlsanity.com/paper/2403.18062/) (2024)
- [HiMemFormer: Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipation](https://mlsanity.com/paper/2411.01455/) (2024)
- [Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior](https://mlsanity.com/paper/2609.17064/) (2026)

## Abstract (arXiv)

This work focuses on anticipating long-term human actions, particularly using short video segments, which can speed up editing workflows through improved suggestions while fostering creativity by suggesting narratives. To this end, we imbue a transformer network with a symbolic knowledge graph for action anticipation in video segments by boosting certain aspects of the transformer's attention mechanism at run-time. Demonstrated on two benchmark datasets, Breakfast and 50Salads, our approach outperforms current state-of-the-art methods for long-term action anticipation using short video context by up to 9%.