mlsanityData to 14 Oct 2026

What is More Likely to Happen Next? Video-and-Language Future Event Prediction

arXiv 2010.07999 ↗First posted 15 Oct 2020cs.CL cs.AI cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

Given a video with aligned dialogue, people can often infer what is more likely to happen next. Making such predictions requires not only a deep understanding of the rich dynamics underlying the video and dialogue, but also a significant amount of commonsense knowledge. In this work, we explore whether AI models are able to learn to make such multimodal commonsense next-event predictions. To support research in this direction, we collect a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event prediction examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips. In order to promote the collection of non-trivial challenging examples, we employ an adversarial human-and-model-in-the-loop data collection procedure. We also present a strong baseline incorporating information from video, dialogue, and commonsense knowledge. Experiments show that each type of information is useful for this challenging task, and that compared to the high human performance on VLEP, our model provides a good starting point but leaves large room for future work. Our dataset and code are available at: https://github.com/jayleicn/VideoLanguageFuturePred

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 17; 2023: 15; 2024: 8; 2025: 7; 2026: 5.

62 papers.

Sort:

By area:

  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  3. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  4. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  5. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  6. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  7. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  8. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  9. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  10. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  11. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  12. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
38 more
  1. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  2. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  3. 2021Human-Adversarial Visual Question Answeringcited by 57
  4. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  5. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  6. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  7. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  8. 2021On the Diversity and Limits of Human Explanationscited by 30
  9. 2022Visual Abductive Reasoningcited by 30
  10. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  11. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  12. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  13. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  14. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  15. 2022When can I Speak? Predicting initiation points for spoken dialogue agentscited by 9
  16. 2024A Survey of Video Datasets for Grounded Event Understandingcited by 9
  17. 2025Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPOcited by 9
  18. 2021Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Cluescited by 8
  19. 2023ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videoscited by 8
  20. 2023Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyondcited by 7
  21. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  22. 2023Modeling Events and Interactions through Temporal Processes -- A Surveycited by 6
  23. 2024Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Modelscited by 6
  24. 2024Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignmentcited by 6
  25. 2023Let's Think Frame by Frame with VIP: A Video Infilling and Prediction Dataset for Evaluating Video Chain-of-Thoughtcited by 5
  26. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMscited by 5
  27. 2022Find Someone Who: Visual Commonsense Understanding in Human-Centric Groundingcited by 4
  28. 2023Commonsense for Zero-Shot Natural Language Video Localizationcited by 4
  29. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4
  30. 2022Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehensioncited by 3
  31. 2024Streaming Detection of Queried Event Startcited by 3
  32. 2021Winning the ICCV'2021 VALUE Challenge: Task-aware Ensemble and Transfer Learning with Visual Conceptscited by 2
  33. 2022Relational Future Captioning Model for Explaining Likely Collisions in Daily Taskscited by 2
  34. 2022Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modelingcited by 2
  35. 2023Personality-aware Human-centric Multimodal Reasoning: A New Task, Dataset and Baselinescited by 2
  36. 2023VCD: Visual Causality Discovery for Cross-Modal Question Reasoningcited by 2
  37. 2024Listen Then See: Video Alignment with Speaker Attentioncited by 2
  38. 2024Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queriescited by 2

50 most cited of 62 All 62 (JSON)