mlsanityData to 7 Oct 2026

CLEVRER: CoLlision Events for Video REpresentation and Reasoning

arXiv 1910.01442 ↗First posted 3 Oct 2019cs.CV cs.AI cs.CLcited by 475

Not on the map. Why?

Cited by 111 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

Key facts. Cited by 475 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 56; 2023: 69; 2024: 87; 2025: 97; 2026: 114.

475 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2024Long Context Transfer from Language to Visioncited by 502
  7. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  8. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  9. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  10. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  11. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  12. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
38 more
  1. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  2. 2021Conditional Object-Centric Learning from Videocited by 247
  3. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  4. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  5. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  6. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 209
  7. 2021Towards Causal Representation Learningcited by 207
  8. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  9. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  10. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  11. 2025Do generative video models understand physical principles?cited by 151
  12. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  13. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  14. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  15. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  16. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  17. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  18. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  19. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  20. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  21. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  22. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  23. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  24. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  25. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  26. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  27. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  28. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  29. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  30. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  31. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  32. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  33. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  34. 2022VIDM: Video Implicit Diffusion Modelscited by 78
  35. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  36. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  37. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  38. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68

50 most cited of 475 All 475 (JSON)