mlsanityData to 14 Oct 2026

Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challenges

arXiv 2609.31665 ↗First posted 15 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Action anticipation predicts future human actions from partial video under incomplete context and temporal uncertainty. Recent systems introduce large language models (LLMs), vision-language models (VLMs), or language-derived semantics at different stages, but reported gains are difficult to interpret when task formulation, visual pretraining, supervision, decoder design, and evaluation code change simultaneously. The central contribution of this review is an evidence-aware design map that crosses task regime with the point at which language-derived information intervenes. We characterise task regimes along six axes. These axes organise the literature into five broad task families: single-action, sequence, object-interaction, cross-view, and planning-oriented settings. C1-C3 locate interventions in context construction, goal/intention modelling, and future decoding, while C4 is treated as an adjacent, emerging grounding/executability extension. Unlike a generic processing pipeline, the map links each intervention to an appropriate counterfactual, failure diagnosis, and permissible evidence claim. Supporting contributions include a protocol-level audit of Ego4D-LTA and EPIC-KITCHENS-100, a multidimensional evidence profile, and the Backbone-Aware Comparison and Ablation Protocol (BCAP). The unresolved EK-100 record is treated as a reporting-comparability case study and is not used as a leaderboard. Evidence for LLM benefits, goal ambiguity, and horizon effects is therefore formulated as testable hypotheses requiring matched validation, not as causal conclusions. The accompanying package contains the coded evidence, source locators, protocol metadata, and versioned catalogue used in the review.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 68 of its 92 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

68 of 92 references.

References · 68

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  6. 2021ViViT: A Video Vision Transformercited by 1,620
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  12. 2017Selective Classification for Deep Neural Networkscited by 852
38 more
  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  3. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  4. 2020Rescaling Egocentric Visioncited by 540
  5. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  6. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  7. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  8. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  9. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  10. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  11. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  12. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  13. 2018Human Action Recognition and Prediction: A Surveycited by 227
  14. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  15. 2023Video Understanding with Large Language Models: A Surveycited by 206
  16. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  17. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  18. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  19. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  20. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  21. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  22. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 92
  23. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
  24. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
  25. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  26. 2026Hierarchical Planning with Latent World Modelscited by 43
  27. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 37
  28. 2022Intention-Conditioned Long-Term Human Egocentric Action Forecastingcited by 35
  29. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 33
  30. 2023PALM: Predicting Actions through Language Modelscited by 29
  31. 2023Object-centric Video Representation for Long-term Action Anticipationcited by 24
  32. 2023Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipationcited by 21
  33. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  34. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  35. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  36. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
  37. 2025Intention-Guided Cognitive Reasoning for Egocentric Long-Term Action Anticipationcited by 10
  38. 2023Technical Report for Ego4D Long Term Action Anticipation Challenge 2023cited by 7

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision and Intention Boost Large Language Model in Long-Term Action Anticipation
  2. 2024From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation
  3. 2026Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
  4. 2025Multi-level and Multi-modal Action Anticipation
  5. 2026The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models
  6. 2023Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023
  7. 2026EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning
  8. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models