# MultiVENT: Multilingual Videos of Events with Aligned Natural Text

arXiv: https://arxiv.org/abs/2307.03153 · HTML: https://mlsanity.com/paper/2307.03153/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 6 Jul 2023.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections](https://mlsanity.com/paper/2205.12005/) (2022)
- [mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video](https://mlsanity.com/paper/2302.00402/) (2023)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [Bridging Video-text Retrieval with Multiple Choice Questions](https://mlsanity.com/paper/2201.04850/) (2022)
- [Cross-media Structured Common Space for Multimedia Event Extraction](https://mlsanity.com/paper/2005.02472/) (2020)
- [Object-aware Video-language Pre-training for Retrieval](https://mlsanity.com/paper/2112.00656/) (2021)
- [A CLIP-Hitchhiker's Guide to Long Video Retrieval](https://mlsanity.com/paper/2205.08508/) (2022)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension](https://mlsanity.com/paper/2305.03347/) (2023)
- [WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus](https://mlsanity.com/paper/2304.04358/) (2023)

## Built on by (most cited first, 7 of 7)

- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [MuMA-ToM: Multi-modal Multi-Agent Theory of Mind](https://mlsanity.com/paper/2408.12574/) (2024)
- [MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval](https://mlsanity.com/paper/2410.11619/) (2024)
- [A Survey of Video Datasets for Grounded Event Understanding](https://mlsanity.com/paper/2406.09646/) (2024)
- [MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion](https://mlsanity.com/paper/2503.20698/) (2025)
- [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025)
- [When Jokes Cross the Line: Analyzing Regular Humor and Dark Humor in YouTube Shorts](https://mlsanity.com/paper/2606.00046/) (2026)

## Abstract (arXiv)

Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage. Datasets that reflect the diverse array of multimodal, multilingual news sources available online could be used to teach models to benefit from this shift, but existing news video datasets focus on traditional news broadcasts produced for English-speaking audiences. We address this limitation by constructing MultiVENT, a dataset of multilingual, event-centric videos grounded in text documents across five target languages. MultiVENT includes both news broadcast videos and non-professional event footage, which we use to analyze the state of online news videos and how they can be leveraged to build robust, factually accurate models. Finally, we provide a model for complex, multilingual video retrieval to serve as a baseline for information retrieval using MultiVENT.