Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges
Not on the map. Why?
Abstract (arXiv)
Action anticipation predicts future human actions from partial video under incomplete context and temporal uncertainty. Recent systems introduce large language models (LLMs), vision-language models (VLMs), or language-derived semantics at different stages, but reported gains are difficult to interpret when task formulation, visual pretraining, supervision, decoder design, and evaluation code change simultaneously. The central contribution of this review is an evidence-aware design map that crosses task regime with the point at which language-derived information intervenes. We characterise task regimes along six axes. These axes organise the literature into five broad task families: single-action, sequence, object-interaction, cross-view, and planning-oriented settings. C1-C3 locate interventions in context construction, goal/intention modelling, and future decoding, while C4 is treated as an adjacent, emerging grounding/executability extension. Unlike a generic processing pipeline, the map links each intervention to an appropriate counterfactual, failure diagnosis, and permissible evidence claim. Supporting contributions include a protocol-level audit of Ego4D-LTA and EPIC-KITCHENS-100, a multidimensional evidence profile, and the Backbone-Aware Comparison and Ablation Protocol (BCAP). The unresolved EK-100 record is treated as a reporting-comparability case study and is not used as a leaderboard. Evidence for LLM benefits, goal ambiguity, and horizon effects is therefore formulated as testable hypotheses requiring matched validation, not as causal conclusions. The accompanying package contains the coded evidence, source locators, protocol metadata, and versioned catalogue used in the review.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 68 of its 92 references that are in the map. First posted on arXiv 15 Sep 2026.
Builds on
68 of 92 references.
References · 68
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
- 2021ViViT: A Video Vision Transformercited by 1,620
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
- 2017Selective Classification for Deep Neural Networkscited by 852
38 more
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
- 2015Sequence Level Training with Recurrent Neural Networkscited by 576
- 2020Rescaling Egocentric Visioncited by 540
- 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
- 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
- 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
- 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
- 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
- 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
- 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
- 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
- 2018Human Action Recognition and Prediction: A Surveycited by 227
- 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
- 2023Video Understanding with Large Language Models: A Surveycited by 206
- 2019Cross-task weakly supervised learning from instructional videoscited by 199
- 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
- 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
- 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
- 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
- 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
- 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 92
- 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
- 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
- 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
- 2026Hierarchical Planning with Latent World Modelscited by 43
- 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 37
- 2022Intention-Conditioned Long-Term Human Egocentric Action Forecastingcited by 35
- 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 33
- 2023PALM: Predicting Actions through Language Modelscited by 29
- 2023Object-centric Video Representation for Long-term Action Anticipationcited by 24
- 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
- 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
- 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
- 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
- 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
- 2025Intention-Guided Cognitive Reasoning for Egocentric Long-Term Action Anticipationcited by 10
- 2023Technical Report for Ego4D Long Term Action Anticipation Challenge 2023cited by 7
50 most cited of 68 All 68 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Vision and Intention Boost Large Language Model in Long-Term Action Anticipation
- 2024From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation
- 2026Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
- 2025Multi-level and Multi-modal Action Anticipation
- 2026The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models
- 2023Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023
- 2026EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning
- 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models