# How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs

arXiv: https://arxiv.org/abs/2405.03690 · HTML: https://mlsanity.com/paper/2405.03690/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 6 May 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)

## Built on by (most cited first, 30 of 30)

- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token](https://mlsanity.com/paper/2501.03895/) (2025)
- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [VLM4D: Towards Spatiotemporal Awareness in Vision Language Models](https://mlsanity.com/paper/2508.02095/) (2025)
- [Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos](https://mlsanity.com/paper/2503.13646/) (2025)
- [Judge Anything: MLLM as a Judge Across Any Modality](https://mlsanity.com/paper/2503.17489/) (2025)
- [MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos](https://mlsanity.com/paper/2506.04141/) (2025)
- [Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs](https://mlsanity.com/paper/2506.07180/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs](https://mlsanity.com/paper/2406.10326/) (2024)
- [VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding](https://mlsanity.com/paper/2412.02186/) (2024)
- [Is Your Video Language Model a Reliable Judge?](https://mlsanity.com/paper/2503.05977/) (2025)
- [Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning](https://mlsanity.com/paper/2501.05069/) (2025)
- [FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compression](https://mlsanity.com/paper/2412.04317/) (2024)
- [CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling](https://mlsanity.com/paper/2602.13191/) (2026)
- [VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment](https://mlsanity.com/paper/2406.10889/) (2024)
- [EgoTrigger: Toward Audio-Driven Image Capture for Human Memory Enhancement in All-Day Energy-Efficient Smart Glasses](https://mlsanity.com/paper/2508.01915/) (2025)
- [A Culturally-diverse Multilingual Multimodal Video Benchmark & Model](https://mlsanity.com/paper/2506.07032/) (2025)
- [Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning](https://mlsanity.com/paper/2512.05513/) (2025)
- [LLaVAction: evaluating and training multi-modal large language models for action understanding](https://mlsanity.com/paper/2503.18712/) (2025)
- [SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs](https://mlsanity.com/paper/2602.22716/) (2026)
- [DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025](https://mlsanity.com/paper/2506.21891/) (2025)
- [Bridging the Gap Between Multimodal Foundation Models and World Models](https://mlsanity.com/paper/2510.03727/) (2025)
- [MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning](https://mlsanity.com/paper/2601.10649/) (2026)
- [Towards Automatic Soccer Commentary Generation with Knowledge-Enhanced Visual Reasoning](https://mlsanity.com/paper/2604.00057/) (2026)
- [Team of One: Cracking Complex Video QA with Model Synergy](https://mlsanity.com/paper/2507.13820/) (2025)
- [Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark](https://mlsanity.com/paper/2603.27259/) (2026)
- [Time-Aware Assistive Navigation](https://mlsanity.com/paper/2609.05596/) (2026)
- [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering](https://mlsanity.com/paper/2609.09973/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have led to the development of Video Large Multi-modal Models (Video-LMMs) that can handle a wide range of video understanding tasks. These models have the potential to be deployed in real-world applications such as robotics, AI assistants, medical surgery, and autonomous vehicles. The widespread adoption of Video-LMMs in our daily lives underscores the importance of ensuring and evaluating their robust performance in mirroring human-like reasoning and interaction capabilities in complex, real-world contexts. However, existing benchmarks for Video-LMMs primarily focus on general video comprehension abilities and neglect assessing their reasoning capabilities over complex videos in the real-world context, and robustness of these models through the lens of user prompts as text queries. In this paper, we present the Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES), a novel benchmark that comprehensively assesses the performance of Video-LMMs across 11 diverse real-world video dimensions. We evaluate 9 recent models, including both open-source and closed-source variants, and find that most of the Video-LMMs, especially open-source ones, struggle with robustness and reasoning when dealing with complex videos. Based on our analysis, we develop a training-free Dual-Step Contextual Prompting (DSCP) technique to enhance the performance of existing Video-LMMs. Our findings provide valuable insights for building the next generation of human-centric AI systems with advanced robustness and reasoning capabilities. Our dataset and code are publicly available at: https://mbzuai-oryx.github.io/CVRR-Evaluation-Suite/.