# VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos

arXiv: https://arxiv.org/abs/2506.02448 · HTML: https://mlsanity.com/paper/2506.02448/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences](https://mlsanity.com/paper/2001.06891/) (2020)
- [Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2212.04500/) (2022)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction](https://mlsanity.com/paper/2211.07342/) (2022)
- [Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video](https://mlsanity.com/paper/1906.02549/) (2019)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models](https://mlsanity.com/paper/2303.13009/) (2023)

## Built on by (most cited first, 8 of 8)

- [AURA: Always-On Understanding and Real-Time Assistance via Video Streams](https://mlsanity.com/paper/2604.04184/) (2026)
- [Video-CoE: Reinforcing Video Event Prediction via Chain of Events](https://mlsanity.com/paper/2603.14935/) (2026)
- [EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction](https://mlsanity.com/paper/2510.21786/) (2025)
- [ClimateCause: Complex and Implicit Causal Structures in Climate Reports](https://mlsanity.com/paper/2604.14856/) (2026)
- [LogiShot: Logically Coherent Cross-Shot Video Generation](https://mlsanity.com/paper/2608.08820/) (2026)
- [NEST: Narrative Event Structures in Time for Long Video Understanding](https://mlsanity.com/paper/2606.19706/) (2026)
- [ShotPlan: Cinematic Video Generation with Learnable Planning Token](https://mlsanity.com/paper/2607.17675/) (2026)
- [Reading Between the Frames of Multimodal Drivelology](https://mlsanity.com/paper/2608.04939/) (2026)

## Abstract (arXiv)

Despite the significant impact of visual events on human cognition, understanding events in videos remains a challenging task for AI due to their complex structures, semantic hierarchies, and dynamic evolution. To address this, we propose the task of video event understanding that extracts event scripts and makes predictions with these scripts from videos. To support this task, we introduce VidEvent, a large-scale dataset containing over 23,000 well-labeled events, featuring detailed event structures, broad hierarchies, and logical relations extracted from movie recap videos. The dataset was created through a meticulous annotation process, ensuring high-quality and reliable event data. We also provide comprehensive baseline models offering detailed descriptions of their architecture and performance metrics. These models serve as benchmarks for future research, facilitating comparisons and improvements. Our analysis of VidEvent and the baseline models highlights the dataset's potential to advance video event understanding and encourages the exploration of innovative algorithms and models. The dataset and related resources are publicly available at www.videvent.top.