# ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models

arXiv: https://arxiv.org/abs/2311.07022 · HTML: https://mlsanity.com/paper/2311.07022/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 84 of its 102 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 30 of 84)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)

## Built on by (most cited first, 26 of 26)

- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision](https://mlsanity.com/paper/2412.14446/) (2024)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?](https://mlsanity.com/paper/2506.05287/) (2025)
- [How and where does CLIP process negation?](https://mlsanity.com/paper/2407.10488/) (2024)
- [VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models](https://mlsanity.com/paper/2504.03970/) (2025)
- [4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding](https://mlsanity.com/paper/2503.17827/) (2025)
- [VRR-QA: Visual Relational Reasoning in Videos Beyond Explicit Cues](https://mlsanity.com/paper/2506.21742/) (2025)
- [GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?](https://mlsanity.com/paper/2507.09491/) (2025)
- [MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeX](https://mlsanity.com/paper/2503.21699/) (2025)
- [VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models](https://mlsanity.com/paper/2505.08455/) (2025)
- [A Culturally-diverse Multilingual Multimodal Video Benchmark & Model](https://mlsanity.com/paper/2506.07032/) (2025)
- [SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation](https://mlsanity.com/paper/2504.05925/) (2025)
- [DeVisE: Behavioral Testing of Medical Large Language Models](https://mlsanity.com/paper/2506.15339/) (2025)
- [All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark](https://mlsanity.com/paper/2502.16989/) (2025)
- [Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?](https://mlsanity.com/paper/2506.14805/) (2025)
- [Bridging the Gap Between Multimodal Foundation Models and World Models](https://mlsanity.com/paper/2510.03727/) (2025)
- [Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence](https://mlsanity.com/paper/2603.13091/) (2026)
- [CV-Probes: Studying the interplay of lexical and world knowledge in visually grounded verb understanding](https://mlsanity.com/paper/2409.01389/) (2024)
- [Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos](https://mlsanity.com/paper/2601.16333/) (2026)
- [ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits](https://mlsanity.com/paper/2609.00505/) (2026)
- [The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMs](https://mlsanity.com/paper/2609.04110/) (2026)
- [From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making](https://mlsanity.com/paper/2609.05149/) (2026)

## Abstract (arXiv)

With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as well as their performance gaps compared to human-level understanding. ViLMA also includes proficiency tests, which assess basic capabilities deemed essential to solving the main counterfactual tests. We show that current VidLMs' grounding abilities are no better than those of vision-language models which use static images. This is especially striking once the performance on proficiency tests is factored in. Our benchmark serves as a catalyst for future research on VidLMs, helping to highlight areas that still need to be explored.