# A Culturally-diverse Multilingual Multimodal Video Benchmark & Model

arXiv: https://arxiv.org/abs/2506.07032 · HTML: https://mlsanity.com/paper/2506.07032/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization](https://mlsanity.com/paper/2305.13091/) (2023)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT](https://mlsanity.com/paper/2402.16840/) (2024)
- [How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs](https://mlsanity.com/paper/2405.03690/) (2024)
- [Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities](https://mlsanity.com/paper/2105.00067/) (2021)
- [ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models](https://mlsanity.com/paper/2311.07022/) (2023)
- [HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation](https://mlsanity.com/paper/2505.11454/) (2025)
- [X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs](https://mlsanity.com/paper/2407.13851/) (2024)
- [Preserving Modality Structure Improves Multi-Modal Learning](https://mlsanity.com/paper/2308.13077/) (2023)
- [VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding](https://mlsanity.com/paper/2403.14743/) (2024)
- [TimeLogic: A Temporal Logic Benchmark for Video QA](https://mlsanity.com/paper/2501.07214/) (2025)

## Built on by (most cited first, 5 of 5)

- [Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses](https://mlsanity.com/paper/2601.13024/) (2026)
- [TowerVision: Understanding and Improving Multilinguality in Vision-Language Models](https://mlsanity.com/paper/2510.21849/) (2025)
- [MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning](https://mlsanity.com/paper/2601.10649/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Large multimodal models (LMMs) have recently gained attention due to their effectiveness to understand and generate descriptions of visual content. Most existing LMMs are in English language. While few recent works explore multilingual image LMMs, to the best of our knowledge, moving beyond the English language for cultural and linguistic inclusivity is yet to be investigated in the context of video LMMs. In pursuit of more inclusive video LMMs, we introduce a multilingual Video LMM benchmark, named ViMUL-Bench, to evaluate Video LMMs across 14 languages, including both low- and high-resource languages: English, Chinese, Spanish, French, German, Hindi, Arabic, Russian, Bengali, Urdu, Sinhala, Tamil, Swedish, and Japanese. Our ViMUL-Bench is designed to rigorously test video LMMs across 15 categories including eight culturally diverse categories, ranging from lifestyles and festivals to foods and rituals and from local landmarks to prominent cultural personalities. ViMUL-Bench comprises both open-ended (short and long-form) and multiple-choice questions spanning various video durations (short, medium, and long) with 8k samples that are manually verified by native language speakers. In addition, we also introduce a machine translated multilingual video training set comprising 1.2 million samples and develop a simple multilingual video LMM, named ViMUL, that is shown to provide a better tradeoff between high-and low-resource languages for video understanding. We hope our ViMUL-Bench and multilingual video LMM along with a large-scale multilingual video training set will help ease future research in developing cultural and linguistic inclusive multilingual video LMMs. Our proposed benchmark, video LMM and training data will be publicly released at https://mbzuai-oryx.github.io/ViMUL/.