# CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models

arXiv: https://arxiv.org/abs/2506.09943 · HTML: https://mlsanity.com/paper/2506.09943/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 33)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [HourVideo: 1-Hour Video-Language Understanding](https://mlsanity.com/paper/2411.04998/) (2024)
- [PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding](https://mlsanity.com/paper/2504.13180/) (2025)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning](https://mlsanity.com/paper/1803.07616/) (2018)
- [ComPhy: Compositional Physical Reasoning of Objects and Events from Videos](https://mlsanity.com/paper/2205.01089/) (2022)
- [Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos](https://mlsanity.com/paper/2312.10300/) (2023)
- [From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering](https://mlsanity.com/paper/2205.14895/) (2022)

## Built on by (most cited first, 30 of 35)

- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [Embodied AI Agents: Modeling the World](https://mlsanity.com/paper/2506.22355/) (2025)
- [PAI-Bench: A Comprehensive Benchmark For Physical AI](https://mlsanity.com/paper/2512.01989/) (2025)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models](https://mlsanity.com/paper/2601.16007/) (2026)
- [How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position](https://mlsanity.com/paper/2606.15032/) (2026)
- [What-If World: A Causal Benchmark for General World Models in Embodied Scenarios](https://mlsanity.com/paper/2605.27589/) (2026)
- [CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning](https://mlsanity.com/paper/2601.13304/) (2026)
- [Counterfactual World Models via Digital Twin-conditioned Video Diffusion](https://mlsanity.com/paper/2511.17481/) (2025)
- [CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models](https://mlsanity.com/paper/2605.23699/) (2026)
- [OpenWorldLib: A Unified Codebase and Definition of Advanced World Models](https://mlsanity.com/paper/2604.04707/) (2026)
- [YoCausal: How Far is Video Generation from World Model? A Causality Perspective](https://mlsanity.com/paper/2605.30346/) (2026)
- [Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs](https://mlsanity.com/paper/2606.05966/) (2026)
- [SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios](https://mlsanity.com/paper/2511.17649/) (2025)
- [Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding](https://mlsanity.com/paper/2511.19923/) (2025)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [Principia: Relational Physics Tests for Video Models](https://mlsanity.com/paper/2609.04200/) (2026)
- [ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering](https://mlsanity.com/paper/2508.21010/) (2025)
- [PhysFire-WM: A Physics-Informed World Model for Emulating Fire Spread Dynamics](https://mlsanity.com/paper/2512.17152/) (2025)
- [Modeling the Mental World for Embodied AI: A Comprehensive Review](https://mlsanity.com/paper/2601.02378/) (2025)
- [Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs](https://mlsanity.com/paper/2605.09422/) (2026)
- [CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding](https://mlsanity.com/paper/2608.30289/) (2026)
- [MultiEcho: An Experimental Science of Learned Worlds](https://mlsanity.com/paper/2609.33347/) (2026)
- [PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop](https://mlsanity.com/paper/2610.00559/) (2026)
- [Spot The Ball: A Benchmark for Visual Social Inference](https://mlsanity.com/paper/2511.00261/) (2025)
- [SCP: Spatial Causal Prediction in Video](https://mlsanity.com/paper/2603.03944/) (2026)
- [ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Models](https://mlsanity.com/paper/2604.16405/) (2026)
- [Act2See: Emergent Active Visual Perception for Video Reasoning](https://mlsanity.com/paper/2605.01657/) (2026)
- [CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering](https://mlsanity.com/paper/2605.23216/) (2026)
- [The Abstraction Gap in Vision-Language Causal Reasoning](https://mlsanity.com/paper/2605.28779/) (2026)

## Abstract (arXiv)

We introduce CausalVQA, a benchmark dataset for video question answering (VQA) composed of question-answer pairs that probe models' understanding of causality in the physical world. Existing VQA benchmarks either tend to focus on surface perceptual understanding of real-world videos, or on narrow physical reasoning questions created using simulation environments. CausalVQA fills an important gap by presenting challenging questions that are grounded in real-world scenarios, while focusing on models' ability to predict the likely outcomes of different actions and events through five question types: counterfactual, hypothetical, anticipation, planning and descriptive. We designed quality control mechanisms that prevent models from exploiting trivial shortcuts, requiring models to base their answers on deep visual understanding instead of linguistic cues. We find that current frontier multimodal models fall substantially below human performance on the benchmark, especially on anticipation and hypothetical questions. This highlights a challenge for current systems to leverage spatial-temporal reasoning, understanding of physical principles, and comprehension of possible alternatives to make accurate predictions in real-world settings.