# Understanding Multimodal Complementarity for Single-Frame Action Anticipation

arXiv: https://arxiv.org/abs/2601.22039 · HTML: https://mlsanity.com/paper/2601.22039/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 64 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 46)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Revealing Single Frame Bias for Video-and-Language Learning](https://mlsanity.com/paper/2206.03428/) (2022)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose](https://mlsanity.com/paper/2007.00394/) (2020)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction](https://mlsanity.com/paper/2002.08945/) (2020)
- [LOKI: Long Term and Key Intentions for Trajectory Prediction](https://mlsanity.com/paper/2108.08236/) (2021)
- [KeyVideoLLM: Towards Large-scale Video Keyframe Selection](https://mlsanity.com/paper/2407.03104/) (2024)
- [Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs](https://mlsanity.com/paper/2005.06582/) (2020)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [Contextual Action Recognition with R*CNN](https://mlsanity.com/paper/1505.01197/) (2015)
- [Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models](https://mlsanity.com/paper/2405.20305/) (2024)
- [ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications](https://mlsanity.com/paper/2010.14742/) (2020)
- [ETRI-Activity3D: A Large-Scale RGB-D Dataset for Robots to Recognize Daily Activities of the Elderly](https://mlsanity.com/paper/2003.01920/) (2020)
- [Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools](https://mlsanity.com/paper/2510.08480/) (2025)
- [Object-centric Video Representation for Long-term Action Anticipation](https://mlsanity.com/paper/2311.00180/) (2023)
- [Balancing Multimodal Training Through Game-Theoretic Regularization](https://mlsanity.com/paper/2411.07335/) (2024)

## Built on by (most cited first, 2 of 2)

- [G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding](https://mlsanity.com/paper/2608.20157/) (2026)
- [Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges](https://mlsanity.com/paper/2609.31665/) (2026)

## Abstract (arXiv)

Human action anticipation is commonly treated as a video understanding problem, implicitly assuming that dense temporal information is required to reason about future actions. In this work, we challenge this assumption by investigating what can be achieved when action anticipation is constrained to a single visual observation. We ask a fundamental question: how much information about the future is already encoded in a single frame, and how can it be effectively exploited? Building on our prior work on Action Anticipation at a Glimpse (AAG), we conduct a systematic investigation of single-frame action anticipation enriched with complementary sources of information. We analyze the contribution of RGB appearance, depth-based geometric cues, and semantic representations of past actions, and investigate how different multimodal fusion strategies, keyframe selection policies and past-action history sources influence anticipation performance. Guided by these findings, we consolidate the most effective design choices into AAG+, a refined single-frame anticipation framework. Despite operating on a single frame, AAG+ consistently improves upon the original AAG and achieves performance comparable to, or exceeding, that of state-of-the-art video-based methods on challenging anticipation benchmarks including IKEA-ASM, Meccano and Assembly101. Our results offer new insights into the limits and potential of single-frame action anticipation, and clarify when dense temporal modeling is necessary and when a carefully selected glimpse is sufficient.